Category Archives: linux

Getting MTP devices to work on Linux: Nook HD(+)

Android devices typically require you to use MTP to read and write files, instead of using USB mass storage. Wondering why? A lot of other people have wondered the same thing. This post (from here, the XDA developer forum) provides an explanation:

“MTP is a big improvement over USB mass storage — for devices with lots of internal memory, a manufacturer no longer needs to come up with some hard partition between the USB mass storage and internal storage. Instead, they are all in one partition, with MTP providing access to the directory of media files that would normally be available through USB mass storage. This means there is no longer a need for apps on SD card for such devices, because what used to be the ‘internal SD card’ is in the same partition as where applications are stored. The storage on your device can be used for either applications or media, depending on what you want to put on it. You aren’t stuck with how much space the manufacturer decided to leave for the two areas.[…Also,] this means that the media storage doesn’t need to be unmounted from Android when it is being accessed through the PC”

Problem is, Linux MTP is a little problematic.  For example, the Nook I bought recently doesn’t work out of the box on my Slackware-current install despite having the latest libmtp – the device turned out to be read-only. Initially I had to use ADB to copy files onto it, but we found a simple solution. If your MTP devices turn out to be read-only on Linux, then there’s a strong possibility that the problem is with libMTP. Specifically, the devices may need to be listed in this file – music-players.h.  For this you need the vendor number and the device ID. To get these, plug the device in via USB and type ‘lsusb’. Find the relevant line:

Bus 001 Device 036: ID 2080:0006 Barnes & Noble

The first of those (2080) is the vendor ID. The second is the device ID. Adding these to libmtp and recompiling resolved the Nook HD issue for me; maybe it will work for you too. Pretty soon libmtp will contain these devices as standard (the HD+ is already in there), but then there’ll be another crop of Android devices with similar MTP problems around in  a minute, so it’s worth remembering the trick anyway.

Installation notes on ipython-notebook

ipython-notebook is a nice piece of kit, a sort of user-friendly pocket web interface slash gui for mucking around with Python. According to the official site, it’s “a web-based interactive computational environment where you can combine code execution, text, mathematics, plots and rich media into a single document”. More introductory information is available here (along with some very pretty screenshots).

I like the idea of it, not so much for development work but for data analysis and worked demonstration. For something to share it certainly beats a command-line session.

It looks rather like this:

python-notebook

I encountered it during an e-Humanities workshop yesterday and I’m already itching to play with it again.

Getting it to install on Slackware involved a few steps. There are probably better ways, but here’s how I did it, based more or less on http://raj.blog.archive.org/2012/02/02/installing-ipython-notebook-to-replace-matlab/:

Step 0: optional. Open sbopkg as root. Search for and install python3. Version 2.6 works with the ipython-notebook but 3 likes sets and UTF8 better.
Step 1: curl -O http://python-distribute.org/distribute_setup.py
Step 2: sudo python distribute_setup.py
Step 3: easy_install pip
Step 4: Now that you have pip, you can follow the latter part of the blog post above:

Open sbopkg as root. Search for and install zeromq.

For the moment, install libpgm-5.1.116~dfsg from source (tar xvfz libpgm-5.1.116~dfsg.tar.gz; ./configure; make; make install).

pip install ipython
pip install pyzmq
pip install tornado
pip install --upgrade ipython
pip install numpy
pip install matplotlib
Follow the configuration instructions described in the blog post.

Accessing local-network-only web pages from outside the firewall

VPNs aren’t as painless as they ought to be, and setting one up purely in order to get at web pages that have been hidden behind a firewall can seem to be overkill sometimes. But then, there are good reasons for hiding things like internal finance-and-admin web pages behind a firewall. What to do?

OpenSSH to the rescue! Greg Tourte just pointed out an alternative to using the VPN in order to access these internal web pages, such as Agresso. It certainly works on Linux and on the Mac, and should also work using Putty (see http://home.fnal.gov/~dwd/ssh-to-browse-behind-firewall.html). This is pretty useful. For example, people within UKOLN who might be having difficulty accessing internal web pages such as Agresso due to VPN problems should be able to use this method instead.

It relies on the fact that OpenSSH has a lot of little-known functionality, in this case, the ability to act as a SOCKS proxy (see http://en.wikipedia.org/wiki/SOCKS for lengthy and boring introduction). In short, OpenSSH can tunnel through to a machine at UKOLN and then allow the browser to treat that connection as a standard web proxy. Because the web proxy endpoint is inside the Bath network, it permits access to internal web pages.

Here is a blog post describing the basics of this openssh functionality:
http://alien.slackbook.org/blog/securely-browsing-the-net-using-socks/

Here are the steps required to set up a SOCKS proxy in order to access these internal web pages remotely (we just tested this on a Mac and on Linux).

1. Install Foxyproxy on Firefox : https://addons.mozilla.org/en-US/firefox/addon/2464

2. Open a terminal window and type: ssh -D 8888 yourusername@InternalServer.YourUniversity.ac.uk
Type in your password when prompted to do so. You will then get a ssh session on InternalServer; just leave this open.

3. Open FoxyProxy, and complete the following steps:

a) add a new manually configured proxy, with the host/IP ‘localhost’, port 8888, click ‘SOCKS proxy?’ and set it to SOCKS v5.

b) Add a new pattern defining when this proxy should be used:
Given the example of setting this up for Bath’s Agresso server:
Call the pattern something like ‘Agresso’, and put in the URL: *agresso.bath.ac.uk/*
Under ‘URL Inclusion/Exclusion’, select ‘Whitelist’, and under ‘Pattern Contains’, select ‘Wildcards’
Make sure that the newly set up proxy is enabled.

 

4. Type http://agresso.bath.ac.uk/ into the address bar. Hopefully, FoxyProxy should make use of the pattern that we have just set up and send the traffic through via the SOCKS proxy.

Caveats: Obviously, just as you need to turn on the VPN before you can access Agresso via the VPN, you will also need to run the SSH step before you can access Agresso via SSH – the SOCKS proxy only lasts for as long as the SSH session remains connected. That said, you can simplify the setup process by adding it to your ~/.ssh/config file, for example:

Host agresso
HostName InternalServer.YourUniversity.ac.uk
DynamicForward 8888
User MyUsername

If I then type ‘ssh agresso’, and type in my password, that does the trick.

This simply goes one step closer to proving that no matter what the problem, if it involves a network, SSH has an answer.

Technical standards

In our project (Writeslike.us) we are using a number of different techniques and tools to make things done.

Coding

As main development languages scripting languages Perl and Python were selected. Python is good for text parsing because language features and external library – Natural language toolkit (NLTK.org) which allow to stem and tag text. The text parsing is non-trivial tasks so for that purpose NLTK use heuristic approach and also offer a training data set for parser to be trained. Perl is native choice for development under Linux and it has powerful set of libraries in CPAN system. In particular XML parsing and full text document crawling functions were written in Perl.
As development tools we are using Eclipse IDE with Perl and Pydev for Perl and Python respectively. In some cases VIM featured text editor is in use. Using such heterogeneous tools implies that project file structure should be flat and simple.

Infrastructure

For running and hosting we are using such proven tools as MySQL and Apache. Subversion (SVN) is used to store and keep track of software versions.

Architecture

The calculation we need to perform to build authors relations network is too resource-intensive to be performed on demand, therefore we need to generated and store database tables with pre-calculated data as some sort of caching approach. Since our project is re-using existing metadata (in Dublin Core (oai-dc, often from qualified DC)) from a repository we were required to add to original project feature list also XML parser and document full text crawler functions.
In order increase usability of the project offered functionality we created a REST interface which enable machine2machine interface. The interface enabled in both directions, i.e. for adding new information about person or publication and for making query to find peers.
Further development plans adding automatic metadata extraction servers to improve quantity and quality of data or to extract further information (FixRep, paperBase).


What programming languages we use and why we love it/them  -technologies, standards, frameworks that make our lives easier (or harder).

In actual fact there are rather a lot of these! In the case of writeslike.us we have stuck to scripting languages, in particular to Perl and Python. Each of these have their benefits and their issues. Arguments centre around Python’s semantic whitespace versus Perl’s line-noise pseudo-ppp transmission ‘write-once, read never’ look, for example. However, both come with a great variety of extensions and libraries. The ‘killer app’ for Python was NLTK, and once one is used to Perl’s CPAN, it becomes indispensable for certain tasks. In the end, arguing about which language is better is pointless, even though it is great fun. The Computer Science department presently teaches Python, and as such Python is the language with which most CS students are more familiar, whilst EE students seem to be either Perl or Matlab according to recent evidence.

In the end, the important point is that prototypes are developed quickly and easily, and that the techniques and datasets underlying them are well understood. If this is the case, then the rest can usually be adapted to suit – rapid development is not the same thing as throwaway prototyping, but rationalisation of software platforms and standards can very well be part of evolutionary prototype enhancement.

USB webcam fun

Bought a pair of webcams (mostly for the purposes of playing with opencv). When plugged in, the uvcvideo kernel module recognises them, and they self-report to the lsusb command as 18ec:3288. However, by default they fail to work (ioctl error).

v4l2: ioctl set format failed: Invalid argument
v4l2: ioctl set format failed: Invalid argument
v4l2: ioctl set format failed: Invalid argument
tv.c: norm_from_string(pal): Bogus norm parameter, setting default.
v4l2: ioctl enum norm failed: Invalid argument
Error: Cannot set norm!
Selected input hasn't got a tuner!
v4l2: ioctl set mute failed: Invalid argument

As is so often true there is an answer for this problem at the linux UVC dev list. Reproducing for the record the wise words of Laurent Pinchard,

Try setting the quirks parameter to 2 before plugging your webcam (either with ‘modprobe uvcvideo quirks=2‘ if the driver is not already loaded, or with ‘echo 2 > /sys/modules/uvcvideo/parameters/quirks‘ if the driver is already loaded).

To make these settings permanent,you need to create the file /etc/modprobe.d/uvcvideo.conf containing the line:

options uvcvideo quirks=2

From then on, the uvcvideo module will always be loaded with the “quirks=2” option and dmesg will show this line when loading the module:

uvcvideo: Forcing device quirks 0x2 by module parameter for testing purpose.
uvcvideo: Please report required quirks to the linux-uvc-devel mailing list.

Linux on a Toshiba NB200

Thanks to Ultim8Fury, Leesa and Nick255 for the following.

Presumably as it’s a fairly new model, after setting up linux (Ubuntu Netbook Remix in this case) on a Toshiba NB200 I ran into a few little problems.  Most of the smaller ones were solved/mitigated by Ultim8Fury’s excellent Setup Guide here.  However if, like me, you had windows off and linux on faster than ** then you may have run into the ‘enabling wifi’ problem.

The drivers suggested by Ultim8Fury cannot turn the card on and windows XP will not reinstall without some very advanced ‘slipstreaming’ technique due to a lack of drivers for the RAID interface.  So if you forgot to enable wireless before removing XP and do not own a copy of Vista (presumably this may work, I don’t actually know) how do you enable the card?

It was Leesa’s explaination of how to enable bluetooth and Nick255’s adaptation of it that provided the answer.

Originally Posted by Leesa
Bluetooth is working with omnibook module:

  1. Get sources from http://dl.getdropbox.com/u/362618/om…1217-1_all.deb
  2. Make sure Bluetooth is enabled in Windows
  3. sudo apt-get install module-assistant build-essential
  4. sudo m-a a-i omnibook-source
  5. Try it: sudo modprobe omnibook ectype=14
  6. Make it autoload:
  • sudo nano /etc/modules
  • Put “omnibook” at the latest line
  • sudo nano /etc/modprobe.d/omnibook.conf:

options omnibook ectype=14 userset=0 lcd=0 display=0 blank=0 battery=0 ac=0 bluetooth=1

Works fine for my NB200, even enabling/disabling bluetooth via /proc/omnibook/bluetooth
You should use ectype=12 instead of ectype=14. That way you can enable or disable wifi without needing to boot Windows. Unfortunately, the hotkeys don’t work, so you have to manually echo either 1 or 0 to /proc/omnibook/wifi (or if you prefer, just disable it in the bios when you want to be sure it is disabled).
To summarise (in Ubuntu):
  1. Download http://dl.getdropbox.com/u/362618/om…1217-1_all.deb
  2. Install the omnibook-source .deb file, module-assistant and build-essential.
  3. Run module-assistant on the omnibook-source package.
  4. Load the module with ectype=12.
  5. Turn the Wifi on and off with /proc/omnibook/wifi

As root:

# This will install any dependencies not already on the system:
apt-get install bzip2 debhelper dpatch kernel-package make module-assistant build-essential
mkdir /tmp/omnibook
cd /tmp/omnibook
wget http://dl.getdropbox.com/u/362618/omnibook-source_2.20070211%2Bsvn20071217-1_all.deb
dpkg -i omnibook-source_2.20070211+svn20071217-1_all.deb
m-a a-i omnibook-source
modprobe omnibook ectype=12
echo 1 > /proc/omnibook/wifi

If this works as advertised, go ahead and add “omnibook” to the end of /etc/modules and “options omnibook ectype=12” to /etc/modprobe.d/omnibook.conf.

To disable the wireless adapter just use the command “echo 0 > /proc/omnibook/wifi”. To enable it, replace the “0” with “1”.