Author Archives: et207

Technical standards

Leave a reply

In our project (Writeslike.us) we are using a number of different techniques and tools to make things done.

Coding

As main development languages scripting languages Perl and Python were selected. Python is good for text parsing because language features and external library – Natural language toolkit (NLTK.org) which allow to stem and tag text. The text parsing is non-trivial tasks so for that purpose NLTK use heuristic approach and also offer a training data set for parser to be trained. Perl is native choice for development under Linux and it has powerful set of libraries in CPAN system. In particular XML parsing and full text document crawling functions were written in Perl.
As development tools we are using Eclipse IDE with Perl and Pydev for Perl and Python respectively. In some cases VIM featured text editor is in use. Using such heterogeneous tools implies that project file structure should be flat and simple.

Infrastructure

For running and hosting we are using such proven tools as MySQL and Apache. Subversion (SVN) is used to store and keep track of software versions.

Architecture

The calculation we need to perform to build authors relations network is too resource-intensive to be performed on demand, therefore we need to generated and store database tables with pre-calculated data as some sort of caching approach. Since our project is re-using existing metadata (in Dublin Core (oai-dc, often from qualified DC)) from a repository we were required to add to original project feature list also XML parser and document full text crawler functions.
In order increase usability of the project offered functionality we created a REST interface which enable machine2machine interface. The interface enabled in both directions, i.e. for adding new information about person or publication and for making query to find peers.
Further development plans adding automatic metadata extraction servers to improve quantity and quality of data or to extract further information (FixRep, paperBase).


What programming languages we use and why we love it/them  -technologies, standards, frameworks that make our lives easier (or harder).

In actual fact there are rather a lot of these! In the case of writeslike.us we have stuck to scripting languages, in particular to Perl and Python. Each of these have their benefits and their issues. Arguments centre around Python’s semantic whitespace versus Perl’s line-noise pseudo-ppp transmission ‘write-once, read never’ look, for example. However, both come with a great variety of extensions and libraries. The ‘killer app’ for Python was NLTK, and once one is used to Perl’s CPAN, it becomes indispensable for certain tasks. In the end, arguing about which language is better is pointless, even though it is great fun. The Computer Science department presently teaches Python, and as such Python is the language with which most CS students are more familiar, whilst EE students seem to be either Perl or Matlab according to recent evidence.

In the end, the important point is that prototypes are developed quickly and easily, and that the techniques and datasets underlying them are well understood. If this is the case, then the rest can usually be adapted to suit – rapid development is not the same thing as throwaway prototyping, but rationalisation of software platforms and standards can very well be part of evolutionary prototype enhancement.

Day-to-day work

Leave a reply

1.scrum meeting + meeting minutes

  1. see the progress made by what people present – what they did yesterday, what they expect to do today, and what worked for them. Allow the manager to keep track of how the project development is progressing and how the team are performing.
  2. find and solve potential problems before they become significant or expensive!

2. collaboration on code

  1. Allows developers to learn from each others’ expertise; permits peer-review of code (extreme programming-style?) and review of functionality.
  2. A long-term goal for this approach is to encourage developers to share code, to think of it as ‘our code’ rather than ‘my code’ and to be more open to review, reuse, constructive criticism, etc.

3. moving code between machines & testing code on different workstations

  1. A single functional installation does not mean that a development project is finished, since it may be very difficult to set up on other platforms, to understand or to reuse.
  2. It should be functionally portable and include all necessary libraries, scripts, datasets and configuration to promote remote development, reuse and external contribution to the codebase.
  3. This also encourages review and testing since it tends to highlight any difficulties with installation and use of newly developed components.

Active collaboration and a flexible approach to development in particular tend to optimise productivity, in that time spent coding also has a knowledge sharing component – and there is relatively little time spent becoming familiar with code before beginning to contribute.

Writeslike.us scenarios

Leave a reply

Title: Looking for a supervisor for a research project
Author: Emma Tonkin and Debra Hiom

Narrative: Alice is a student at the University of Oxford. She is looking for a supervisor for her MSc level research in Information Science. She knows what sort of research area she wants to work in, and she has found several example papers of interest to her. She would like to be able to use those papers as a first step in looking for researchers working in that area who either work at her university, an affiliated institution or somewhere geographically local. She would like to get back a list of researchers, their institutional and departmental roles and their previous work and supervisory experience.

Title: Classifying events and forums by listed participants
Author: Em (adapted from Ana’s Tea For Two article, D-Lib)

Narrative: Jonathan is a researcher in evolutionary linguistics. He has become very interested in possible mathematical mechanisms for describing the nature, growth and adaption of language, as he has heard that others, such as Partha Nyogi, have done some very interesting work in this area. Unfortunately, Jonathan is not a mathematician and finds that some of the detail is hard to follow. He realises that what he really needs to do is either to go to the right sort of event or the right sort of online forum and find some people who might be interested in exploring links between his specialist area and their own. Both of these are difficult in their own ways. To go to the right sort of event would mean identifying what sort of event that would be, and he does not have enough money to go to very many. So he chooses to look up possible events and web forums, thinking that he can look through the participant lists for names that he recognises. This is greatly simplified by a system that uses information about the papers and authors that he considers most relevant; with this information it is able to parse through lists of participants in events or online communities in order to provide him with a rough classification of how relevant the group is likely to be to his ideas.

Title: Building a ‘dance card’ for an ISRC event
Author: Em

Narrative: One of the purposes of an ISRC (Information Sciences Research Council, a fictitious but plausible organisation) event is to encourage the serendipitous meeting. Rather than simply assuming that synchronicity at the coffee-table will carry the day, the ISRC decide to produce a ‘dance card’ that suggests several other individuals that you might like to meet. Whilst elements of the composition of this ‘dance card’ are resultant from program managers’ knowledge of the individual’s interests and character, the writeslike.us service can be used to quickly build some interesting (and at times amusing) meeting suggestions, based on the individuals’ papers and output, and/or on the names and ISRC-held descriptions of the projects on which the individuals work.

Title: Facilitating collaboration in a multidisciplinary research environment
Author: Em

Narrative: Ben is an anthropologist with a particular interest in the area of paleolithic archaeology. He works in the Department of Humanities. He is very interested in exploring likely patterns of migration, and particularly in the idea that this activity may have been driven by climate change. However, the Department of Humanities has limited funding for the purpose of data collection and interpretation regarding modeling of climate change, so it is not possible for him to develop a paleoclimate simulation system. Therefore he decides that it is more appropriate for him to look for other people who have other reasons to be interested in modeling of this kind, particularly during the time period in which he is interested. This is not a trivial problem for several reasons; firstly, he does not usually publish in the same area as paleoclimatologists and therefore is unlikely to make chance acquaintances. Secondly, he and they have very different ways of describing their areas of interest, and therefore there is quite a lot of interpretation required in order to ascertain that the datasets they require are (or are not) closely related.

Title: Recording collaboration; figuring out what worked best
Author: Em

Narrative: Jennifer is a program manager for the Information Sciences Research Council, a fictitious but plausible organisation that funds information sciences research in the UK. She is tasked with deciding the future direction of ISRC funding for research-related events and related online community grants in the UK. In order to facilitate the decision, she decides that it is necessary to seek some evidence able to characterise the profile of past exemplars of these events, and the current state of the online groups that they fund. Her analysis shows that certain events attract a broad and diverse set of individuals from organisations all over the UK, and give rise to unexpected and interdisciplinary collaboration, whereas certain others attract fairly closed cliques of similarly focused individuals who are also linked online by Twitter accounts and other mechanisms for informal communication. This aids her in making her eventual decision regarding which events should receive greater funding.

SWOT Analysis: writeslike.us

Leave a reply

Strengths

Good knowledge of data source aggregation and normalisation
Agile rapid prototype development and evaluation
Links to IEMSR, FixRep, Aggregator: internal resources – strong support network

Weaknesses
Staff time is limited
Various programming languages in use

Opportunities
Lots of interest in auto-extraction of community networks data
University of Minho now have funding to take work forward
University of Leiden interest in specific technical tagging
Learning from NAMES and enhancing other existing services like NAMES
FixRep stuff: data provenance, evaluation, quality assurance
Establishment of user community

Threats
Reliance on external services/data
Quality of source data and availability of source data is variable
User community not firmly established

USB webcam fun

Leave a reply

Bought a pair of webcams (mostly for the purposes of playing with opencv). When plugged in, the uvcvideo kernel module recognises them, and they self-report to the lsusb command as 18ec:3288. However, by default they fail to work (ioctl error).

v4l2: ioctl set format failed: Invalid argument
v4l2: ioctl set format failed: Invalid argument
v4l2: ioctl set format failed: Invalid argument
tv.c: norm_from_string(pal): Bogus norm parameter, setting default.
v4l2: ioctl enum norm failed: Invalid argument
Error: Cannot set norm!
Selected input hasn't got a tuner!
v4l2: ioctl set mute failed: Invalid argument

As is so often true there is an answer for this problem at the linux UVC dev list. Reproducing for the record the wise words of Laurent Pinchard,

Try setting the quirks parameter to 2 before plugging your webcam (either with ‘modprobe uvcvideo quirks=2‘ if the driver is not already loaded, or with ‘echo 2 > /sys/modules/uvcvideo/parameters/quirks‘ if the driver is already loaded).

To make these settings permanent,you need to create the file /etc/modprobe.d/uvcvideo.conf containing the line:

options uvcvideo quirks=2

From then on, the uvcvideo module will always be loaded with the “quirks=2” option and dmesg will show this line when loading the module:

uvcvideo: Forcing device quirks 0x2 by module parameter for testing purpose.
uvcvideo: Please report required quirks to the linux-uvc-devel mailing list.

Get your Java code automatically checked against Anti-patterns

Leave a reply

What could be better than good and clean Java code written following Anti-patterns recommendations?

Only code which is automatically check and every anti-pattern piece of code is flagged. It is incredibly easy to achieve just by adding a Eclipse plug-in to your IDE, assuming of course that Eclipse is your development platform.

Writeslike.us project meeting to generate ideas about the project approach

Leave a reply
As a result of a recent meeting we have come up with the following  ideas in one way or another related to the Writeslike.us project. The meeting was all about ‘name disambiguation’.
First we discussed names: does the existence of two documents by an author or authors with the same name, but published in different places, indicate that there are two people with the same name? Or does this represent one author publishing with two publishers?
2 publications from one person - does it mean 2 people (with the same name) or it is one person?

2 publications from one person - does it mean 2 people (with the same name) or it is one person?

The problem is unsolvable without additional information identifying the person. But for the case when this information is unavailable, there is a suggestion to apply pure statistics. In other words, evaluate for a testing (sample) data set what error level is linked to both possibilities – two different persons or one, just publishing in two places.

Then we switched to looking at practical methods to use within the Writeslike.us project, to identify individuals who write similarly or about similar topics.

Person identity dimensions + table organization for mapping

Person identity dimensions + table organization for mapping

This lovely diagram displays how the system we’re building will do the ‘magic’. Since none of the dimensions available to us are able to discriminate identity alone, we need to bring several onto the stage. The result will be an approximation incorporating evidence from several sources, which will hopefully make it more precise.

The table is just a representation of the way in which the heap of raw data will be mapped into something useful.

Welcoming Wei Jiang to writeslike.us

Leave a reply

A new staff member (casual staff) has just begun work, specifically on the Writeslike.us project. He is very experienced in designing and implementing web portals, familiar with machine learning concepts and enthusiastic about project research area – constructing a graph of relations between researchers based on analysis of their papers.

Here is his home page

http://wei-jiang.com

Created a mind map for the Writeslike.us project direction

Leave a reply

Here is a mind map of system architecture and idea. How to read it:

  • From the Repo extract the metainformation
  • save it to the internal database
  • then use a process to determine, who else is in this author’s community?
  • the process may have a variety of interfaces (a gadget for Repository Interface – a list of authors in the same community; or it could be search engine saying: Authors from the same community also wrote those papers)

Documents are analysed by a number of dimensions (could be weighted) like used keywords, bibliographical references and social tags.

Here is a data model, let’s say documents are related to others by those dimensions (draw with different colours in the diagram), then the process calculate relation ‘value’ by browsing particular document links.