dimanche 25 novembre 2012

Booting the Android ICS system from SDCard

Update: from the information below, I recently added the "recovery" partition, so it shifts partition numbers.
The mele flash kitchen_v2 contains scripts to generate the sdcard version, look here:
Mele A2000: my own Linux flash kitchen

Testing new firmwares could be a pain since everything is wiped out at each update.
My idea (an I see some people had the same idea before) is to write the Android partitions on sdcard and boot from it.

There are already Linux distributions that boot from SDCard, like ArchLinux:
I have taken this Linux distribution as the base system.

The principle is quite easy: take a SDCard u-boot, give it the Android kernel, create the Android partitions on SDCard, fill them to make Android happy, change all "nandx" references by "mmcblk0px" in init scripts.

The Android system boots and does not need any access to NAND, a good point since I can test firmwares without affecting my flashed one.


Under linux, I have partitioned the SDCard (SDHC class 10, for boot speed) with fdisk.
Here is the resulting partitions:

Note that a free space of 16MB is kept at start of sdcard, it embeds the spl bootloader
I put 4GB for "data" and 10GB for /sdcard (16GB SDCard gives lot of space).
Exact size doesn't matter; if you set to 15,95MB one partition, that's not a big issue. The important thing is the number of partitions, see below.

So the matching nand-sdcard partitions are:

-nand- -sdcard-   -name-      -format-
nanda  mmcblk0p1  bootloader  vfat
nandb  -N/A-      env
nandc  mmcblk0p2  boot        ext4
nandd  mmcblk0p3  system      ext4
nande  mmcblk0p5  data        ext4
nandf  mmcblk0p6  misc        ext4
nandg  -N/A-      recovery
nandh  mmcblk0p7  cache       ext4
nandi  mmcblk0p8  private     vfat 
nandj  -N/A-      sysrecovery
nandk  mmcblk0p9  UDISK       vfat

Note: I just found out that link: http://linux-sunxi.org/Android/partitions
Almost same thing...
Note(2): under linux, mmcblk0p1 is /dev/sdb1, mmcblk0p2 is /dev/sdb2, ...

Filling partitions

Then I formatted partitions using mkfs.vfat for FAT partitions, and mkfs.ext4 for ext4 partitions that need to be formatted (look at the above table for the ones that need to be formatted, the others will be formatted by Android).
Then, from ArchlinuxARM/Mele-Bootloader I did:
# write SPL
dd if=sun4i-spl.bin of=$DRIVE bs=1024 seek=8
# write mele u-boot
dd if=mele-u-boot.bin of=$DRIVE bs=1024 seek=32
where DRIVE is /dev/sdb is sdcard is on sdb.

In the bootloader partition, I put the Mele's script.bin, and the Mele's kernel (the 3.0.8+ one), transformed into a u-boot image using this:
mkimage -A ARM -C none -T kernel -O linux  -a 40008000 -e 40008000 -d zImage uImage
I added uEnv.txt:
boot_mmc=fatload mmc 0 0x43000000 ${fexfile}; fatload mmc 0 0x48000000 ${kernel}; bootm 0x48000000
extraargs=rootwait init=/init mac_addr=00:BA:D0:MA:C0:FF

The "boot" partition contains the initrd contents of root.fex, and must be in mmcblk0p2 (default root partition of the spl u-boot, you can change it with root=xxx in extraargs variable)

The "system" partitions contents the files of the /system partition (system.fex mounted partition)
That's all.

Changing init scripts

In "boot" partition, you need to edit init.sun4i.rc and uevent.sun4i.rc, to change all "nandX" references into "mmcblk0pN" references conforming to the above table.
grep -r nand * shows you all the files that need to be modified.

In the same manner, change /system:

The proof

Here is the mount table from my functional system:
root@android:/ # busybox mount                                                                                                      
rootfs on / type rootfs (rw)                                                                                                        
/dev/root on / type ext4 (ro,relatime,user_xattr,barrier=1,data=ordered)                                                            
devtmpfs on /dev type devtmpfs (rw,relatime,size=156796k,nr_inodes=39199,mode=755)                                                  
tmpfs on /dev type tmpfs (rw,nosuid,relatime,mode=755)                                                                              
devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)                                                                  
proc on /proc type proc (rw,relatime)                                                                                               
sysfs on /sys type sysfs (rw,relatime)                                                                                              
tmpfs on /mnt type tmpfs (rw,relatime,mode=777,gid=1000)                                                                            
tmpfs on /mnt/asec type tmpfs (rw,relatime,mode=755,gid=1000)                                                                       
tmpfs on /mnt/obb type tmpfs (rw,relatime,mode=755,gid=1000)                                                                        
/dev/block/mmcblk0p3 on /system type ext4 (ro,relatime,user_xattr,barrier=0,data=ordered)                                           
/dev/block/mmcblk0p5 on /data type ext4 (rw,nosuid,nodev,noatime,user_xattr,barrier=0,journal_checksum,data=ordered,noauto_da_alloc)
/dev/block/mmcblk0p7 on /cache type ext4 (rw,nosuid,nodev,noatime,user_xattr,barrier=0,journal_checksum,data=ordered,noauto_da_alloc
/dev/block/mmcblk0p8 on /mnt/private type vfat (rw,relatime,fmask=0000,dmask=0000,allow_utime=0022,codepage=cp437,iocharset=ascii,sh
/dev/block/vold/179:1 on /mnt/extsd/179_1 type vfat (rw,dirsync,nosuid,nodev,noexec,relatime,uid=1000,gid=1023,fmask=0702,dmask=0702
/dev/block/vold/179:6 on /mnt/extsd/179_6 type vfat (rw,dirsync,nosuid,nodev,noexec,relatime,uid=1000,gid=1023,fmask=0702,dmask=0702
/dev/block/vold/179:8 on /mnt/extsd/179_8 type vfat (rw,nosuid,nodev,noexec,relatime,fmask=0000,dmask=0000,allow_utime=0022,codepage
/dev/block/vold/179:9 on /mnt/extsd/179_9 type vfat (rw,dirsync,nosuid,nodev,noexec,relatime,uid=1000,gid=1023,fmask=0702,dmask=0702

And the storage screenshot:

mercredi 7 novembre 2012

Handling nand flash to save/recover data using ArchLinuxArm

When booting from SDCard under linux, it is possible to access the NAND partitions.
I use ArchLinux ARM, following this tutorial: Mele A100, the only bootable image that gets correctly the NAND partitions (as under Android ICS). I changed the MAC entry in script.bin in order to have a well-known MAC address for ssh.

Under Android ICS, partitions are defined like this:

-dev-  -name-        -start-       -size-       -mountpoint-

nanda  bootloader  : 1000000       01000000      u-boot
nandb  env         : 2000000       01000000      N/A (u-boot environment)
nandc  boot        : 3000000       02000000      initrd in RAM as / + kernel
nandd  system      : 5000000       20000000      /system
nande  data        : 25000000      60000000      /data
nandf  misc        : 85000000      01000000      "boot-recovery" or 0xff*
nandg  recovery    : 86000000      02000000      N/A (recovery image)
nandh  cache       : 88000000      08000000      /cache
nandi  private     : 90000000      01000000      /mnt/private
nandj  sysrecovery : 91000000      20000000      /mnt/sysrecovery
nandk  UDISK       : b1000000      3b000000      /sdcard

Under Linux, partitions are detected correctly (the following line shows something comparable to the above output):

echo "-dev-  -size-";cat /proc/partitions|while read x y size dev; do echo "$dev  `printf %08x $(( $size * 1024 ))`";done
-dev-  -size-
nanda  01000000
nandb  01000000
nandc  02000000
nandd  20000000
nande  60000000
nandf  01000000
nandg  02000000
nandh  08000000
nandi  01000000
nandj  20000000
nandk  3c800000

As a consequence, you can read/edit/save data from NAND using a linux distribution.
Important note: don't write on Android ICS mounted partitions, you can corrupt data!
Use cases:
1/ list installed application in /data:
mount -o noatime,nosuid,nodev,barrier=0,journal_checksum,noauto_da_alloc /dev/nande /mnt #this is how Android mounts ext4 partitions, look at init.sun4i.rc for special nand mount options.
ls -l /mnt/app/
-rw-r--r-- 1 1000 1000  4707663 Nov  5  2012 com.adobe.flashplayer-1.apk       

2/ backup a NAND partition locally:
dd if=/dev/nande bs=4096 |bzip2 -c >/root/nande.bz2

3/ backup a NAND partition via ssh (assuming my ssh server is at @
dd if=/dev/nande bs=4096 |bzip2 -c | ssh user@ dd of=/home/user/nande.bz2

4/ restore a NAND partition locally:
bzip2 -cd /root/nande.bz2 | dd of=/dev/nande bs=4096

5/ restore a NAN partition via ssh:
ssh user@ bzip2 -cd /home/user/nande.bz2 | dd of=/dev/nande bs=4096

6/ make a tar.bz2 archive of a partition contents (for huge partitions, instead of saving raw data):

mount -o noatime,nosuid,nodev,barrier=0,journal_checksum,noauto_da_alloc /dev/nande /mnt
cd /mnt
tar jcf - .|ssh user@ dd of=/home/user/nande.tar.bz2
umount /mnt

Using these commands, you can save/restore completely some essential partitions before a crash or a new firmware flashing procedure (/data essentially, /sdcard contains user data - movies, ... - that you should save in another place by other means).

dimanche 4 novembre 2012

Creating a dd-able image (Mele_4.0_V1.3BETA_revised_en_us_4.img.bz2)

I saw the lundman method (see conversation: http://ibot.rikers.org/%23arm-netbook/20120702.html.gz, 03:01.59):
- dd if=/dev/zero of=/dev/sdx bs=4096 count=128000 #an image does not exceed 512MB, and I take 4096 as blocksize for read/write flash speedup.
- flash the firmware on the sdcard using PhoenixCard,
- using fdisk -l /dev/sdx you know the FAT partition begin, and as as consequence, the hidden partition end.
- hexdump the /dev/sdx -s <at supposed end of last partition>, see when it gets some zero, you have the real end of the flash image.
Example, on Mele_4.0_V1.3BETA_revised_en_us_4.img
Result: Mele_4.0_V1.3BETA_revised_en_us_4.ddimg.bz2
I flashed my firmware on my sdcard, that is shown as /dev/sdb.

fdisk -l /dev/sdb shows (I have a 2GB  SDCard):

Disk /dev/sdb: 1967 MB, 1967128576 bytes
1 heads, 62 sectors/track, 61968 cylinders, total 3842048 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *      695166     3839998     1572416+   b  W95 FAT32

Start of the visible partition is at 695166 sector # (means at 695166*512=355924992 bytes):

xxd -a -skip 355924992 /dev/sdb
1536fc00:eb3c 904d 5344 4f53 352e 3000 0208 2000  .<.MSDOS5.0... .
1536fc10:0200 0000 00f0 0000 0000 0000 0000 0000  ................
1536fc20:81fc 2f00 fd0b 0000 0000 0000 0200 0000  ../.............
1536fc30:0100 0600 0000 0000 0000 0000 0000 0000  ................
1536fc40:8000 296f 0000 0020 2020 2020 2020 2020  ..)o...         
1536fc50:2020 4641 5433 3220 2020 0000 0000 0000    FAT32   ......
1536fc60:0000 0000 0000 0000 0000 0000 0000 0000  ................
1536fdf0:0000 0000 0000 0000 0000 0000 0000 55aa  ..............U.
1536fe00:0000 0000 0000 0000 0000 0000 0000 0000  ................
15373c00:f8ff ffff ffff ffff ffff ff0f 0000 0000  ................
15373c10:0000 0000 0000 0000 0000 0000 0000 0000  ................
154f3600:f8ff ffff ffff ffff 0000 0000 0000 0000  ................
154f3610:0000 0000 0000 0000 0000 0000 0000 0000  ................
15673000:2020 2020 2020 2020 2020 2008 d0da e18c             .....
15673010:6441 6441 0000 e18c 6441 0000 0000 0000  dAdA....dA......
15673020:0000 0000 0000 0000 0000 0000 0000 0000  ................
1576fc00:f8ff ffff ffff ff0f ffff ff0f 0000 0000  ................
1576fc10:0000 0000 0000 0000 0000 0000 0000 0000  ................
15943600:f8ff ffff ffff ff0f ffff ff0f 0000 0000  ................
15943610:0000 0000 0000 0000 0000 0000 0000 0000  ................
15b17000:2020 2020 2020 2020 2020 2008 0000 0000             .....
15b17010:0000 0000 0000 0000 0000 0000 0000 0000  ................
... and nothing more until a little time.

Remember: I zeroed 4096*128000 bytes, I must find out the end, where there are many zeroes until this limit (4096*128000=0x1f400000).
In this case: 0x15b17010 seems to be the end of my FAT32 partition.

So you can suppose that the end of image (including the fat32 partitions) is at 0x15b17010.
0x15b17010 makes in decimal: 88856 blocks of 4096 bytes (rounded up).

dd if=/dev/sdb bs=4096 count=88856 |bzip2 -9 > Mele_4.0_V1.3BETA_revised_en_us_4.ddimg.bz2
The resulting file is a compressed dd-able image.

To write this dd-able image, you can perform:
bzip2 -cd Mele_4.0_V1.3BETA_revised_en_us_4.ddimg.bz2|dd of=/dev/sdx bs=4096

Mele_HTPC_0929_BETA_V1.3 Revised version: en_US v4

Another release that fixes the "reboot never ends" bug after having launched Google Play:

On 4shared:

Edit: md5sum=70aee0c6ae9c375e72d3f87bed594ef0  Mele_4.0_V1.3BETA_revised_en_us_4.img 
        dd-able image: Mele_4.0_V1.3BETA_revised_en_us_4.ddimg.bz2
        md5sum=ea14dab60e131feac04f20a7947291e9  Mele_4.0_V1.3BETA_revised_en_us_4.ddimg.bz2

Precision: this release disables the Google Play updates automatically (via a /etc/init.d/ script), that makes the bug occur. You don't need to perform the steps mentioned hereafter if you take it.

I hope this will be the last under Android 4.0 (next will be 4.1 JellyBean when Mele decides to release it).
After many tries (copy Vending.apk from old firmware, revert all /system/app and /system/framework, ...), nothing solved the "reboot never ends" issue.

Then I had the idea to revert the Market update (go to Paremeters->Applications->All->"Google Play"->"Uninstall updates".
And yes, it works. The Market update is what makes the system fail.

So the solution is to disable the Market update definitely. I don't know the consequences on a long term period, but for this release, no more "reboot never ends" issue.
In order to disable the Market update, see this thread: (How-to) prevent Android Market app from updating itself, that says:

Do "uninstall updates" on the market app (in "manage applications").
Open the terminal app.
Run "su" to become root.
Run: touch /data/app/com.android.vending-1.apk
Run: chattr +i /data/app/com.android.vending-1.apk

You can perform only "uninstall updates" step, but you will have to do that at each reboot.

In the above image, there is a init.d script that performs the required touch+chattr operations automatically.
I also removed the last Chinese application (sorry for them).

Note for Chinese people: this fix should apply on the Chinese firmware, but you have to add these lines in the init.rc script (on post-fs-data) since there is no init.d system in the original firmware.