Category Archives: code

Code snippets, mini-projects, hacks go here. This is main front page content area 2

Getting MTP devices to work on Linux: Nook HD(+)

Android devices typically require you to use MTP to read and write files, instead of using USB mass storage. Wondering why? A lot of other people have wondered the same thing. This post (from here, the XDA developer forum) provides an explanation:

“MTP is a big improvement over USB mass storage — for devices with lots of internal memory, a manufacturer no longer needs to come up with some hard partition between the USB mass storage and internal storage. Instead, they are all in one partition, with MTP providing access to the directory of media files that would normally be available through USB mass storage. This means there is no longer a need for apps on SD card for such devices, because what used to be the ‘internal SD card’ is in the same partition as where applications are stored. The storage on your device can be used for either applications or media, depending on what you want to put on it. You aren’t stuck with how much space the manufacturer decided to leave for the two areas.[…Also,] this means that the media storage doesn’t need to be unmounted from Android when it is being accessed through the PC”

Problem is, Linux MTP is a little problematic.  For example, the Nook I bought recently doesn’t work out of the box on my Slackware-current install despite having the latest libmtp – the device turned out to be read-only. Initially I had to use ADB to copy files onto it, but we found a simple solution. If your MTP devices turn out to be read-only on Linux, then there’s a strong possibility that the problem is with libMTP. Specifically, the devices may need to be listed in this file – music-players.h.  For this you need the vendor number and the device ID. To get these, plug the device in via USB and type ‘lsusb’. Find the relevant line:

Bus 001 Device 036: ID 2080:0006 Barnes & Noble

The first of those (2080) is the vendor ID. The second is the device ID. Adding these to libmtp and recompiling resolved the Nook HD issue for me; maybe it will work for you too. Pretty soon libmtp will contain these devices as standard (the HD+ is already in there), but then there’ll be another crop of Android devices with similar MTP problems around in  a minute, so it’s worth remembering the trick anyway.

Archiving Old Dynamic Websites

Archiver is a web based application written in Java using the Spring Framework. Its main use is to produce static versions of websites which are no longer updated with new content. This is important for many reasons; firstly, dynamic sites will typically target specific software dependencies.  As time passes certain dependencies will no longer receive patches, posing a security risk, or updates, meaning it may no longer function with the latest operating systems and code environments. This is almost always a problem for system administrators who sometimes end up maintaining software that can be tens of years old on outdated systems. Archiver offers an easy means of creating a static versions of of these dynamic sites that can deployed simply to a standard web server, typically, but not limited to, Apache.

 

Websites will run using different software; some will be written in plain HTML and CSS whilst others will run on CMS or WIKI platforms such as WordPress, MediaWiki or Drupal. Each of these methods will provide a slightly different way of performing some tasks, writing certain elements in HTML/CSS etc and laying out structure. After analysing the problem it was clear that we would need to target specific software in order to provide a high quality solution. For this reason, the ‘Strategy’ design pattern was chosen.

In this case an interface and super implementation provided a default set of methods for dealing with the processing of individual web elements written to work in the majority of cases. It can be thought of as standard web behavior. Subclasses of this Strategy were provided to account for software differences.

We currently support the following strategies -:

  • Basic (Static websites for migration)
  • Drupal
  • Media Wiki
  • WordPress

One of the main tasks which Archiver performs is to make any links which appear in HTML, CSS or JavaScript files relative to the homepage of the website so they are not absolute links. The JSoup plugin for Java was especially useful in this case as it allows the detection of a specified tag in the HTML file. JSoup also uses a Jquery type syntax to select the different elements from the HTML e.g. “#” is used to select an ID and “.” is used to select a class. JSoup also allows invalid HTML which is useful doesn’t prevent a site from being fully archived if there are mistakes in the markup. For the CSS and JavaScript, Regex was used to create expressions in the specified format for a CSS or JavaScript link, this could then be used to find and change the links. Alongside making links relative, Archiver also adds each link which it finds to the list of files to be added into the archive folder. After archiving recursively a zip file is served up to the user.

While existing solutions are available none of them provide the comprehensive rewriting capabilities of Archiver. All the user has to do is point the webapp at a site, choose a strategy and deploy the resulting zip.

Archiver also produces a README file which provides details of all the files which have been included in the archive and lists any errors such as missing pages.

Code is available from https://bitbucket.org/ukolnisc/archiver/src

While this is working code it has not received sufficient testing which is obviously vital for this type of project. With that in mind we would love to hear your feedback.

 

Detecting motion: DLP-TILT 2-axis accelerometer

Using an accelerometer to assess musculoskeletal stresses

Accelerometers are helpful for characterising device usage and detecting the mode of use. Most of these devices have an internal accelerometer, but it is convenient for our purposes to use an external device (requires less programming and does not require a jailbroken device).

Setup

In order to use this device, you must first set up the serial port with a line like the following:

stty -F /dev/ttyUSB0 raw ispeed 38400 ospeed 38400 cs8 -ignpar -cstopb -echo

Code

The code probably won’t tell you all that much, but here’s one way to make your life much easier: assuming you’re running this on Linux, set your device ID as /dev/dlptilt, then change the line ‘/dev/ttyUSB0’ to read ‘/dev/dlptilt’ in the code before compiling.

#include
#include
#include
#include
#include
#include
#include
#include 

#define BAUDRATE B38400
#define MODEMDEVICE "/dev/ttyS1"
#define _POSIX_SOURCE 1         //POSIX compliant source
#define FALSE 0
#define TRUE 1
#define DEBUG 0

volatile int STOP=FALSE;

void signal_handler_IO (int status);    //definition of signal handler
int wait_flag=TRUE;                     //TRUE while no signal received
char devicename[80];
long Baud_Rate = 38400;         // default Baud Rate (110 through 38400)
long BAUD;                      // derived baud rate from command line
long DATABITS;
long STOPBITS;
long PARITYON;
long PARITY;
int Data_Bits = 8;              // Number of data bits
int Stop_Bits = 1;              // Number of stop bits
int Parity = 0;                 // Parity as follows:
                  // 00 = NONE, 01 = Odd, 02 = Even, 03 = Mark, 04 = Space
int Format = 4;
FILE *input;
FILE *output;
int status;

main(int Parm_Count, char *Parms[])
{

   int fd, tty, c, res, i, error;
   char In1, Key;
   struct termios oldtio, newtio;
   struct termios oldkey, newkey;
   struct sigaction saio;
   char buf[255];
   char message[90];

   // set device name here
   strcpy(devicename,"/dev/ttyUSB0"); 

   newkey.c_cflag = BAUDRATE | CRTSCTS | CS8 | CLOCAL | CREAD;
   newkey.c_iflag = IGNPAR;
   newkey.c_oflag = 0;
   newkey.c_lflag = 0;
   newkey.c_cc[VMIN]=1;
   newkey.c_cc[VTIME]=0;
   BAUD=38400;
   DATABITS=CS8;
   STOPBITS=CSTOPB;
   PARITYON=0;
   PARITY=0;
   fd = open(devicename, O_RDWR | O_NOCTTY | O_NONBLOCK);
   if (fd < 0)
   {
   	perror(devicename);
   	exit(-1);
   }

   saio.sa_handler = signal_handler_IO;
   sigemptyset(&saio.sa_mask);
   saio.sa_flags = 0;
   saio.sa_restorer = NULL;
   sigaction(SIGIO,&saio,NULL);

   fcntl(fd, F_SETOWN, getpid());
   fcntl(fd, F_SETFL, FASYNC);

   tcgetattr(fd,&oldtio);
   newtio.c_cflag = BAUD | CRTSCTS | DATABITS | STOPBITS | PARITYON | PARITY | CLOCAL | CREAD;
   newtio.c_iflag = IGNPAR;
   newtio.c_oflag = 0;
   newtio.c_lflag = 0;
   newtio.c_cc[VMIN]=1;
   newtio.c_cc[VTIME]=0;
   tcflush(fd, TCIFLUSH);
   tcsetattr(fd,TCSANOW,&newtio);
   write(fd,"T",1);
   while (STOP==FALSE) {
       status=1;
       if (status==1) // so redundant...
		{
	if(DEBUG){
   		write(fd,"P",1);
		printf("Printed Pn");
	} else {
		// 7 - single a d conversion to host on current settings
		// z 3 bytes (both channels x and y of accelerometer)
		// s - 7 bytes (all 7 channels)
		// 8 - streaming a/d conversion data using current settings
   		write(fd,"z",1);
	}
	} 

         if (wait_flag==FALSE)  //if output is available
	{
		res = read(fd,buf,255);
		if (res>0)
		{

			// Print serial output
			for (i=0; i

CSS3 – Better Late Than Never

CSS 1/2

Back in the year 2000, we, and by that I mean anyone who wrote web-pages, were bungling around with CSS. I was being told tables were evil, I need to separate code (content) and style and that CSS was the answer to my prayers. While the second was clearly something to aim for, it became obvious that CSS at the time wasn’t really capable of helping me achieve this. Don’t get me wrong, it was a step in the right direction and rightfully consigned laying out sites in tables as a thing of the past. We were encouraged to lay elements out as divs and most of the time it was possible to get at least most of the way into true separation between style and code, for static sites at least.

Dynamic Sites

One problem I repeatedly encountered was when rendering out lists of elements of undetermined size into a tabular style. Maybe I wanted odd elements as a different colour, or my boss might come in and want the last element in cornflower blue to match his Tuesday tie. It was mostly possible but was a pain and tied the code into the visuals in several places.

CSS3 to the rescue

Finally eleven years later CSS3 comes to the rescue with some fancy new selectors (there were a few in CSS2). Here are some good ones I have used already.

This will add ‘>>’ at the end of any external links automatically. Repeat for https if required.

a[href^="http://"]:not([href*="www.domain.com"]):after { 
    content: " >>"; 
}

This will highlight any links that open in a new window.

a[target^="_blank"] { 
    color:#855; 
}

 

It also allows you to write code that is almost (other than the class name) completely detached from the visuals. This lays out divs in columns of three.

/*Loop in your language of choice*/ {
Content
}
.splitThird {height: auto; clear: both; }

.splitThird div:nth-child(3n+1) {
    float:left; width: 33%; clear:left; margin-bottom: 20px;
}

.splitThird div:nth-child(3n+2) {
    float: left; width: 33%; clear: none;
}

.splitThird div:nth-child(3n+3){
    float: right; width: 33%;
}

A Starter’s Guide to the Android SDK (on Linux)

It’s fairly easy to setup and install the Android SDK on Linux. For starters, the fact that you don’t need a Mac, and that it’s freely available really helps. The Android developers website is very resourceful and has a lot of information and step-by-step guides to have the SDK setup and ready in no time. Chances are, you won’t have to look at any other websites for this purpose.

Things You’ll Need

1. Make sure your system meets the system requirements (it’s a few hundred MBs of hard drive storage).

http://developer.android.com/sdk/requirements.html

2. The SDK Starter Package

http://developer.android.com/sdk/index.html

3. ADT Plug-in for Eclipse (if developing in Eclipse). Requires Eclipse 3.5 (Galileo) or greater.

http://www.eclipse.org/downloads/

If not developing in Eclipse, you can use other IDEs like Apache Ant 8.1 or greater (http://ant.apache.org/) or JDK 5 or JDK 6 (http://www.oracle.com/technetwork/java/javase/downloads/index.html)

Installing the SDK

Firstly, the downloaded file must be unzipped.

Zip file Type ‘unzip filename.zip’ in the terminal

.tgz file Type ‘tar zxf filename.tgz’ in the terminal

Once all the files are extracted, eclipse is ready to be used. Typing in ‘eclipse’ in the terminal should start the IDE.

Installing the ADT (Android Development Toolkit) to Eclipse

Follow this link for a step-by-step guide to adding the ADT Plug-in to Eclipse.

http://developer.android.com/sdk/eclipse-adt.html#installing

Install Android Platforms, APIs and other recommended tools

Once the ADT is installed, you can launch the Android Manager from Eclipse.

Open Eclipse

Select Window > Android SDK and AVD Manager.

Select the required components from the Android SDK Manager and it will install the required components for you.

That’s it! You’re ready to start developing your own Android Applications.

 

 

WebFont

Font support in browsers has always been a bit hit and miss. While support is improving, the only real way to have nice crisp fonts, particularly in the larger sizes, is to alias your own images in your graphics editor of choice and embed them. This is very time consuming. For this reason I created WebFont that mimics many paid services and allows inclusion of fonts on a page by using a dynamic url. It can be embedded into existing projects or run as a standalone service.
[box type=”shadow”]View the source at bitbucket[/box]

This snippet is used to fetch the ‘152’ on the RepUK homepage.

As this is updated dynamically it is something that would be impossible to recreate by any other means.

JQuery Plugin: Polling with timeouts

Polls a server resource at specified timeout interval until abortTimeOut is reached with options for testing responses for success.

This was useful for allowing a machine a certain amount of time to wake from hibernate before aborting.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
 /** 
 * PollingWithTimeouts - Polls a server resource at specified timeout 
 * interval until abortTimeOut is reached
 *
 * Settings -:
 * url -          the url to ajax call
 * method -       get/post (defaults to get)
 * sendData -     array of values to be passed in - 
 *                e.g. {name: "foo", something: "else"}
 * timeout -      timeout in ms between each poll (default 1000)
 * type -         json/text/xml what you are expecting as a 
 *                response (default json)
 * abortTimeOut - how long in total before we completely give up in 
 *                milliseconds (default 60000) to have no abort 
 *                timeout dont pass in abortCallBack or testCallBack
 *
 *
 * Usage -:
 *
 * 1) As a never ending poll
 * $(document).ready(function(){
 *       $.PollingWithTimeouts({
 *              url : '/foo/',
 *              sendData:  {foo: 'foo', etc: 'etc'}
 *       });
 * })
 *
 * 2) As a never ending poll with a pollCallBack (stock ticker for example)
 * $(document).ready(function(){
 *       $.PollingWithTimeouts({
 *              url : '/foo/',
 *              sendData:  {foo: 'foo', etc: 'etc'}
 *       },
 *       function(pollCallBackData) { alert(pollCallBackData); }
 *   );
 * })
 *
 * 3) As a poll with an (optional) pollCallBack that aborts after the 
 * specified time calling back to the (optional) abortCallBack
 * 
 * $(document).ready(function(){
 *       $.PollingWithTimeouts({
 *              url : '/foo/',
 *              sendData:  {foo: 'foo', etc: 'etc'},
 *              abortTimeOut: 60000
 *       },
 *       function(pollCallBackData) { alert(pollCallBackData); },
 *       function(abortCallBackData) { alert(abortCallBackData); },
 *   );
 * })
 *
 * 4) As a poll with a call back that aborts after the specified time 
 * calling back to the abort function isCompletedCallBackData allows the 
 * caller to test response data and return true if polling is complete
 *    If isCompletedCallBack is true it calls the successCallBack
 *    e.g. (and the reason I wrote it) after doing WOL on 
 * machine poll the server to ping and determine if the machine is awake. 
 * This needs to abort after a period of time.
 * $(document).ready(function(){
 *     $.PollingWithTimeouts({
 *              url : '/foo/',
 *              sendData:  {foo: 'foo', etc: 'etc'},
 *              abortTimeOut: 60000
 *     },
 *     function(pollCallBackData) { alert(pollCallBackData); },
 *     function(abortCallBackData) { alert(abortCallBackData); },
 *     function(isCompletedCallBackData) {alert(isCompletedCallBackData);},
 *     function(successCallBackData) { alert(successCallBackData); },
 *   );
 * })
 *
 * Copyright (c) 2011 UKOLN (http://www.ukoln.ac.uk)
 * Mark Dewey
 * Licensed under GPL:
 * http://www.gnu.org/licenses/gpl.html
 *
 * Version: 0.9
 */
 
(function($) {
    $.PollingWithTimeouts = function(options, pollCallBack, 
                      abortCallBack, isCompletedCallBack, successCallBack) {
 
            settings = jQuery.extend({
                url: '',        // URL of ajax request
                method: 'get', // get or post
                sendData: '',   // array of values to be passed in 
                                //e.g. {name: "foo", something: "else"}
                timeout: 1000,   // timeout in milliseconds - 1 sec
                type: 'json', //or whatever else you like
                abortTimeOut: 60000  //1 min
            }, options);
 
                f = settings.method == 'post' 
                     || settings.method == 'POST' ? $.post : $.get;
 
                var abort = new Date().getTime() + settings.abortTimeOut;
                getdata();
 
                function getdata()
        {
 
                        f(settings.url, settings.sendData, function(d){
 
                if (abortCallBack &amp;&amp; 
                          new Date().getTime() &gt; abort) {
                        clearTimeout(PollingWithTimeouts);
                        abortCallBack(d);
                }
                else {
                        if (isCompletedCallBack) {
                                if (isCompletedCallBack(d)) {
                                        clearTimeout(PollingWithTimeouts);
                                        if (successCallBack) {
                                                successCallBack(d);
                                        }
                                } else {
                                        if (pollCallBack) {
                                                pollCallBack(d);
                                        }
                                        PollingWithTimeouts 
                                           = setTimeout(getdata, 
                                               settings.timeout);
                                }
                        } else {
                                if (pollCallBack) {
                                        pollCallBack(d);
                                }
                                PollingWithTimeouts = setTimeout(getdata, 
                                                         settings.timeout);
                        }
                }
            }, settings.type)
        }
 
        };
})(jQuery);

Extended Repository PDF Assessment

As part of FixRep a small project is being carried out to examine the use of metadata in pdf documents held in HE/FE repositories (such as the University of Bath’s Opus repository). This builds on an initial pilot that was carried out using pdfs harvested from Opus, which we wrote a paper about for QQML 2010 (Hewson & Tonkin, 2010).

The original study of Opus was an exploration to test out the extraction and analysis process. Obviously the initial analysis focusing on only one repository could only be used to draw conclusions about what’s in Opus; issues it may present, metadata usage, etc. The extended assessment is examining pdfs from about a dozen UK repositories so that a reasonable analysis of metadata, comparison of ‘standard’ usage, and common vs. unique issues, can be obtained.

So, how are we going about this?

It’s a pretty manual process at the moment, at least each of the stages is kicked-off manually, and can be divided into three stages…

  • Harvest the pdf files
  • Extract the metadata into database
  • Analyse content

Harvest…

Using wget the repository structure containing the pdf files is copied to a local server. This process takes some time and can be rather heavy-handed in the overhead it places on the repository server through continual requests for files. If we wanted to extend the extraction and analysis process into a service that regularly updates, then a more considerate approach towards the repositories would be required. However, we’ve got away with it at least this far!

Extract & load…

A prototype service that extracts information from pdf documents was developed as part of the FixRep project. It extracts the information in a number of ways:

Heading and formatting analysis, such as:

  • The version of the PDF standard in use
  • Whether certain features, such as PDF tagging, are declared to be in use
  • The software used to create the PDF
  • The publisher of the PDF
  • The date of creation and last modification

Information about the body of the documents:

  • Whether images or text could be successfully extracted from the document and, if they could, information about those data objects.
  • If any text could be extracted from the object, further information such as the language in which it appeared to be written and the number of words in the text

Information from the originating filesystem, such as:

  • document path
  • document size
  • creation date, etc.

The extracted information is put into intermediate files in an XML format and is then loaded into a MySQL database for…

Analysis…

PDF Processing

PDF Processing

The first thing we actually look at is how many of the harvested pdf files could be processed, and for those that failed, what was the reason they failed. For example in out pilot run against the Opus content about 80% of pdf files could be processed. The 20% that failed were mainly due to the service being unable to extract meaningful information, while a very small number of files turned out to be ‘bad’ pdfs – that is the format of the files was corrupted or not in a recognisable file format. some of the errors identified were recoverable with some manual intervention, while some meant file had to be excluded as un-processable.

While not definitive, this does give us a baseline expectation for the success rate of extracting meaningful information from other repositories.

Once we have the data in the database it’s easy enough to run some sql to generate simple statistics such as: Type and number of distinct metadata tags used;  Average number of metadata tags per file; etc. This gives us a good overview of the content (in a given repository) and whether the content is consistent within and between repositories.

Next steps…

The target repositories have been harvested and will soon be processed for analysis. So, unless some very unexpected processing problems happen we should have results and be ready to produce a report on this project in early December.