Kirkwood U-Boot and Debian Installer Script (WORK-IN-PROGRESS)

bodhi Wrote:
-------------------------------------------------------
> bobafetthotmail Wrote:
>
> > I was talking about LEDE firmware, but yes,
> also
> > the script.
>
> Ah, put the flashing stuff in a single function in
> the script and I'll look it over for you. Is it
> ubifs problem?

No, it is LEDE build system that does not generate ubifs images until I figure out how to configure that.
I made a OpenWRT Chaos Calmer image manually (imagebuilder, --> kirkwood generic target + some necessary packages then unpack the rootfs.tar.gz it generates and packed into ubifs by command line myself, then I added dtb to uimage as usual) and it works on my nsa310, but I wanted to offload that to LEDE.

Anyway if I can't figure it out in a reasonable timescale I'm just uploading a pre-made generic rootfs image + device-specific uimage and the script will install that as recovery.

bobafetthotmail Wrote:
-------------------------------------------------------
> I made a OpenWRT Chaos Calmer image manually
> (imagebuilder, --> kirkwood generic target + some
> necessary packages then unpack the rootfs.tar.gz
> it generates and packed into ubifs by command line
> myself, then I added dtb to uimage as usual) and
> it works on my nsa310, but I wanted to offload
> that to LEDE.
>
I am interested in the above hack. Can you provide the instruction on how to do this?

habibie Wrote:
-------------------------------------------------------
> bobafetthotmail Wrote:
> --------------------------------------------------
> -----
> > I made a OpenWRT Chaos Calmer image manually
> > (imagebuilder, --> kirkwood generic target +
> some
> > necessary packages then unpack the
> rootfs.tar.gz
> > it generates and packed into ubifs by command
> line
> > myself, then I added dtb to uimage as usual)
> and
> > it works on my nsa310, but I wanted to offload
> > that to LEDE.
> >
> I am interested in the above hack. Can you provide
> the instruction on how to do this?

this for using imagebuilder https://wiki.openwrt.org/doc/howto/obtain.firmware.generate

this for making ubifs manually
https://forum.openwrt.org/viewtopic.php?id=63714

Thanks.

> this for making ubifs manually
> https://forum.openwrt.org/viewtopic.php?id=63714

That was a quite good tutorial. But please do take your time to read and understand because it is not for beginners.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

(yes I know I should place this in its own thread in the rescue builds section, will do after testing this)

quick update, found out how to enable ubifs build, but to make an ubifs there is too much device-specific info, we would have to keep dozens of different rootfs images.

The current plan for recovery is making a generic squashfs rootfs with all tools and filesystem drivers we might need, and then make device-specific uImages by appending the dtb to it.
This will be a rescue system based off LEDE, not a true LEDE firmware, so most LEDE features (package manager to download and install new packages, web interface) are removed.

I also plan to place a script in autostart so that when you log in there is a list of common actions you can have the recovery execute on its own.
Like:

     _________
    /        /\      _    ___ ___  ___
   /  LE    /  \    | |  | __|   \| __|
  /    DE  /    \   | |__| _|| |) | _|
 /________/  LE  \  |____|___|___/|___|                      lede-project.org
 \        \   DE /
  \    LE  \    /  -----------------------------------------------------------
   \  DE    \  /    %N (%C, %R)
    \________\/    -----------------------------------------------------------


Welcome to Kirkwood recovery! What do you want to do?
1 check filesystems
2 make rootfs (with options)
3 update uboot
4 make RAID rootfs
5 stuff1
6 stuff2
7 exit

I built a test image (from source, not from imagebuilder) from source because there seems to be no friggin gptfdisk nor parted in their repos (even if their kernel has GPT enabled so it can read such disks), and these are available only in an old third-party repo. And I can add third party repos only if I build the whole thing. >.>

Seriously, no gptfdisk in default repos.... :-/

Parted fails to build (how unexpected), gptfdisk failed too but was easy to fix (add a dependency in makefile), can do without parted.

The squashfs rootfs is 14MB in size, the uimage is 2MB, I assume this is enough to fit in the flash of all devices apart from the thinclient, am I right?

Will have to test this later, if it works I'm opening a thread in the rescue system section.



Edited 1 time(s). Last edit at 08/28/2016 03:45PM by bobafetthotmail.

bobafetthotmail,

I've reviewed kirkwood-installer-12. Thanks for the great job writing this script :) nicely structured. I think it is in the final form, except for a few last comments I have below.


1. Need to check errors or tell the user to stop and post the log, even though we've checked for bad blocks before. Does errno work here? I have not looked at these tools source code so not sure if they set the returned code to non-zero.

# Write new uBoot to mtd0
  # Erase the first 512k
  flash_erase "$DEV_MTD0" 0 4
  #need to add a check on the output here.

  nandwrite "$DEV_MTD0" "$TOOLS/$UBOOT_MTD0"
  #need to add a check on the output here.

2. When we verify md5, tell the user to stop and post the log if it fails here 2nd time

verify_md5 "$TOOLS/new_uboot_dump" "$TOOLS/new_uboot_dump.md5"
  if [ "$?" != "0" ]; then
    rm -f "$TOOLS/new_uboot_dump.md5"
    echo "##"
    echo "##"
    echo "## VERIFICATION FAILED!"
    echo "##"
    echo "## uBoot was not properly installed to mtd0."
    echo "##"
    echo "##"
    echo "## YOUR DEVICE MAY BE IN AN UNUSABLE STATE."
    echo "## DO NOT REBOOT OR POWER OFF YOUR DEVICE"
    echo "##"
    echo "##"
    echo "## Make a backup of /tmp/uboot-mtd0-dump someplace safe and"
    echo "## then re-run this installer."
    exit 1
  else
    echo "# Verified successfully!"
  fi
  rm -f "$TOOLS/new_uboot_dump.md5"

Same for u-boot env, when we verify md5, tell the user to stop and post the log if it fails here 2nd time

# Verify the uBoot environment
  nanddump --noecc --omitoob -f "$TOOLS/uboot.environment.dump" -s 0xc0000 -l 0x20000 "$DEV_MTD0"
  wget -O "$TOOLS/$UBOOT_ENV.md5" "$UBOOT_ENVIRONMENT_IMG_MD5" --no-check-certificate
  verify_md5 "$TOOLS/uboot.environment.dump" "$TOOLS/$UBOOT_ENV.md5"
  if [ "$?" != "0" ]; then
    rm -f "$TOOLS/$UBOOT_ENV.md5"
    echo "## VERIFICATION FAILED!"
    echo "## uBoot environment was not properly written to mtd0.  Please re-run this installer."
    exit 1
  fi
  rm -f "$TOOLS/$UBOOT_ENV.md5"
  rm -f "$TOOLS/uboot.environment_dump"

3. In preserve_crucial_envs_for_later(), mtdparts will need to be adjusted to have orion_nand. Stock u-boots uses mtd_nand, which is not good.

# Preserve the mtdparts
....
  read_env_stock_uboot "mtdparts"
 ...
    read_env_stock_uboot "bootargs"
....
 
 ENV_MTDPARTS="$VALUE"   ### perhaps mtd_nand can be replaced here with orion_nand
  VALUE=""

Example:

mtdparts=orion_nand:0x100000(uboot),0x80000(uboot_env),0x80000(key_store),0x80000(info),0xA00000(etc),0xA00000(kernel_1),0x2FC0000(rootfs1),0xA00000(kernel_2),0x2FC0000(rootfs2)

4. I see choices for the plugs needs be here, so "Pogoplug" will be replaced by "Dockstar", and then adding more devices. I'll add a new device to it and then run a test.

install_rootfs_and_uboot
	;;
	3) #Pogoplug
    echo "disabled"
	exit 1
		PLATFORM="Pogoplug"
		STOCK_MTD0="/dev/mtd0" #this is the how the mtd0 is, as seen from inside the stock firmware
		ENV_DTB_FILE="'/boot/dts/kirkwood-dockstar.dtb'"

		#device-specific commands to be run before proceding
		# Stop the pogoplug engine
		killall -q hbwd

		install_rootfs_and_uboot

	;;

5. I'm not sure if e2label can be used reliably (might not be). Wish I can think of a better indicator. Perhaps checking both e2label and root label = rootfs ?

if [ -f "/sbin/e2label" ]; then
# this means we are inside a Debian rootfs,
# labelling it now as rootfs will screw up mounting later
# it will be labelled later when unmounted
mke2fs -t "$FILESYSTEM" "$PART_1"
else
#not in Debian rootfs, so we can give it a label only this way
mke2fs -t "$FILESYSTEM" -L "rootfs" "$PART_1"
fi

6. Is no-rootfs) a command line option is to install u-boot only? I thought so, just want to doublecheck.

################################################
#####
##### MAIN
#####
################################################

#doing a root check, because the installer must be run as root for obvious reasons
if [ "$(id -u)" -eq 0 ]; then
	echo ;
else
	echo "you must run this script as root";
exit
fi

##loading advanced options passed as arguments, and changing relevant variables

CUSTOM_PARAM=""
MAKE_ROOTFS="YES"

#filesystem to use when formatting
FILESYSTEM="ext3"

for i in "$@"
do
case $i in
    -GPT)
      PARTITION_TABLE="GPT"
    ;;
    -ext4)
      FILESYSTEM="ext4"
    ;;
    -systemd)
    CUSTOM_PARAM="$CUSTOM_PARAM"" init=/bin/systemd"
    ;;
    -no-rootfs)
      MAKE_ROOTFS="NO"
    ;;
    *)
            # unknown options are ignored
    ;;
esac

7. Lastly, I think we should collect telemetry to local text file only, not ftp server. And ask users to zip it and upload it to this forum. But they need to upload it only if (after visit the the forum or checking the GitHub README file and see that we still need the info):

- The device is not on the list of supporting devices, or
- It's on the list of devices that we need to have telemetry.

IMO, people are going to be sensitive if they upload any collected data to any server :) We know it's transparent, it is a script after all (but this won't be obvious to noobs). But things like MAC Address need to be scrubbed from the text file ad we would want to give them a chance to remove all identifiable stuff like that before uploading the file here. I can also add my GitHub email address to the Main banner part of the script so that they can send it somewhere if the forum is not available. And they can do the same when you fork it a second time to make this into a LEDE installation (if you include your email there). In fact, let me write the preamble for this script when it's running and let you look at it.

#uploading to ftp server
ftpput -v -u "anonymous" -p "" "alby.ftp.sh" "/uploads/$ARCHIVE_NAME.bz2" "$ARCHIVE_NAME.bz2"
if [ $? = "0"  ]; then
  cd || return 1
else
#ftp does not seem to work reliably, so we offer to save also locally the file.
....

echo

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 09/05/2016 09:16PM by bodhi.

bodhi Wrote:
-------------------------------------------------------
> 1. Need to check errors or tell the user to stop
> and post the log, even though we've checked for
> bad blocks before. Does errno work here? I
> have not looked at these tools source code so not
> sure if they set the returned code to non-zero.
>

Will do. I didn't yet check that, I focused mostly on getting everything to work at all first.

I also wanted to do a dmesg check for bad blocks in bad places before flashing, so I could adjust the flashing properly

>
> 2. When we verify md5, tell the user to stop and
> post the log if it fails here 2nd time

ok, will add.


> Same for u-boot env, when we verify md5, tell the
> user to stop and post the log if it fails here 2nd
> time

ok.

> 3. In preserve_crucial_envs_for_later(), mtdparts
> will need to be adjusted to have
> orion_nand. Stock u-boots uses
> mtd_nand, which is not good.
>

I noticed this too. When I wanted to flash test LEDE recovery image, I had to change that manually for the kernel to see mtd partitions correctly. There are other envs with orion_nand too in your uboot.

Why that? Is it just a name used in other envs too or is it hardcoded somewhere?

Anyway it's easy to replace with awk, as the mtdparts line always has a ":" that I can use to split the line and drop the part we don't want.


>
> 4. I see choices for the plugs needs be here, so
> "Pogoplug" will be replaced by "Dockstar", and
> then adding more devices. I'll add a new device to
> it and then run a test.

yes, copy the nsa310 entry and change the variables so it can work for the other box.

You will have to check manually if there are drive automount daemons to kill or other such stuff that can get in the way of installing, and either write them as in the Pogoplug entry or make a function like I did for zyxel boxes (as they all use the same userspace in their firmware, the main daemon even states it was developed for nsa220 when it starts up)

>
> 5. I'm not sure if e2label can be used reliably
> (might not be). Wish I can think of a better
> indicator. Perhaps checking both e2label and root
> label = rootfs ?
>

e2label is a tool in the e2fsprogs package on Debian https://packages.debian.org/sid/sh4/e2fsprogs/filelist which is crucial to any installation so I think it is reasonable to assume that if the box is Debian we will have an e2label binary in there.

I don't care if the script will also react to stock firmwares with e2label binary, if they have it we can use it there too and it will be better.

The only issue is if we are in a Debian system with a rootfs labelled "rootfs", stock firmwares don't have partitions with label "rootfs" (or with labels at all) afaik.

But yes, this is a workaround to the fact that there is no reliable way to get what block device is /

>
> 6. Is no-rootfs) a command line option is
> to install u-boot only? I thought so, just want to
> doublecheck.
>

Yes, it is checked inside the install_rootfs_and_uboot() function, and if it is set to "NO" the rootfs will not be installed.

It's a fast way to get the script to install uboot only, I might rework it into something better later.

>
> 7. Lastly, I think we should collect telemetry to
> local text file only, not ftp server.

heh, it seems my home ISP is blocking any ftp traffic from ports that aren't 21. So I can't use passive ftp, so people's firewalls block the connection as active ftp looks like an inbound connection for them. :/

Will have to drop that feature, but I still think it would have been neat.

> And ask
> users to zip it and upload it to this forum. But
> they need to upload it only if (after visit the
> the forum or checking the GitHub README file and
> see that we still need the info):
>
> - The device is not on the list of supporting
> devices, or
> - It's on the list of devices that we need to have
> telemetry.

Well, it's not part of any install process in the script.
That's in its own function called by a "gather telemetry" command inside a "for advanced users" menu.
So yeah, I doubt people can trigger that unless they are very curious or someone asks for it (readme or forum).

> MAC Address need to be scrubbed from the text file
> ad we would want to give them a chance to remove
> all identifiable stuff like that before uploading

I can do that too, but most people don't know what can be sensitive info in the dumps anyway so it will not be very useful for them unless I can automate the edits.

Those that want can open the archive and look at the text file themselves before uploading, it's just a big text file with output of various commands, no encryption no signatures no nothing.
Besides, it's also anonymous if sent to a server instead of posted in a forum.

Consider that they are installing your uboot and your rootfs in binary form without doing big checks on them anyway.
So they are trusting you already to not be an NSA sockpuppet.

Now you could make the point that the ftp server was mine and that *I* could be a NSA sockpuppet... uhhhhh.... I can't confirm or deny that. :)

> the file here. I can also add my GitHub email
> address to the Main banner part of the script so
> that they can send it somewhere if the forum is
> not available.

Good idea. Seconded.

> I noticed this too. When I wanted to flash test
> LEDE recovery image, I had to change that manually
> for the kernel to see mtd partitions correctly.
> There are other envs with orion_nand too in your
> uboot.
>
> Why that? Is it just a name used in other envs too
> or is it hardcoded somewhere?
>

The modern Linux kernels use orion nand driver, unlike older squeeze kernels. So MTD NAND driver was set as Orion in the kernel config. That name is used to find the mtds.

> Anyway it's easy to replace with awk, as the
> mtdparts line always has a ":" that I can use to
> split the line and drop the part we don't want.
>

Sounds good!

> You will have to check manually if there are drive
> automount daemons to kill or other such stuff that
> can get in the way of installing, and either write
> them as in the Pogoplug entry or make a function
> like I did for zyxel boxes (as they all use the
> same userspace in their firmware, the main daemon
> even states it was developed for nsa220 when it
> starts up)
>

OK.

> But yes, this is a workaround to the fact that
> there is no reliable way to get what block device
> is /

I'm thinking about newer boxes that we don't have support yet. They use kernel 3.2.x. So yes I think "rootfs" is what we really care about.

> Will have to drop that feature, but I still think
> it would have been neat.
>
> > And ask
> > users to zip it and upload it to this forum.
> But
> > they need to upload it only if (after visit the
> > the forum or checking the GitHub README file
> and
> > see that we still need the info):
> >
> > - The device is not on the list of supporting
> > devices, or
> > - It's on the list of devices that we need to
> have
> > telemetry.
>
> Well, it's not part of any install process in the
> script.
> That's in its own function called by a "gather
> telemetry" command inside a "for advanced users"
> menu.
> So yeah, I doubt people can trigger that unless
> they are very curious or someone asks for it
> (readme or forum).
>

Understood. I was suggesting that you can put that notice in the script so users know what to do.

> > MAC Address need to be scrubbed from the text
> file
> > ad we would want to give them a chance to
> remove
> > all identifiable stuff like that before
> uploading
>
> I can do that too, but most people don't know what
> can be sensitive info in the dumps anyway so it
> will not be very useful for them unless I can
> automate the edits.

I don't think we need to either (scrub it in the scrip). Privacy-sensitive people will know to what to edit in the log.

>
> Those that want can open the archive and look at
> the text file themselves before uploading, it's
> just a big text file with output of various
> commands, no encryption no signatures no nothing.
> Besides, it's also anonymous if sent to a server
> instead of posted in a forum.

True. But there are people who use VPN for this forum if they really care to be anonymous :)

>
> Consider that they are installing your uboot and
> your rootfs in binary form without doing big
> checks on them anyway.
> So they are trusting you already to not be an NSA
> sockpuppet.

:))

> Now you could make the point that the ftp server
> was mine and that *I* could be a NSA sockpuppet...
> uhhhhh.... I can't confirm or deny that. :)

:))) and you did not receive any letter either :))

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

> The squashfs rootfs is 14MB in size, the uimage is
> 2MB, I assume this is enough to fit in the flash
> of all devices apart from the thinclient, am I
> right?

64MB is the smallest Kirkwood I've seen (there could be smaller one that I dont know about but I think 32MB assumption is safe for router).

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

> I also wanted to do a dmesg check for bad blocks
> in bad places before flashing, so I could adjust
> the flashing properly

One more thing. This is more complicate than just simply bailing out after seeing a bad block in the first 1M, telling them to get help in the forum. Because it might not even be possible to make it work if there is more than 1 bad block. And it won't work if the bad block is at 0xc0000.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 2 time(s). Last edit at 09/06/2016 05:01AM by bodhi.

I just posted a beta of a recovery firmware based off LEDE, in the recovery section, can you sticky it (act quickly, before it's covered by the massive activity in that section :) )

I should be able to post an updated version of the script with what you asked tomorrow, then I'll start writing the modules to install the recovery.

bodhi Wrote:
-------------------------------------------------------
>
> I'm thinking about newer boxes that we don't have
> support yet. They use kernel 3.2.x. So yes I think
> "rootfs" is what we really care about.
>

As I said there is no reliable way of autodetecting the device of / so the script has no way to know if the rootfs is actually / (an issue) or if it is not / (not an issue).

How about kernel version? If the script detects kernel version 3.17 or later (I'm more for 4.0 or later) then it assumes Debian. Otherwise it assumes it is in a firmware.

Is there something in Debian I can call and it will answer "yes I'm Debian"? Maybe kernel says Debian in name.

bobafetthotmail Wrote:
>
> Is there something in Debian I can call and it
> will answer "yes I'm Debian"? Maybe kernel says
> Debian in name.

Debian will have the file /etc/debian_version which will contain the actual version number.

Ray

bobafetthotmail Wrote:
-------------------------------------------------------
> I just posted a beta of a recovery firmware based
> off LEDE, in the recovery section, can you sticky
> it (act quickly, before it's covered by the
> massive activity in that section :) )

Good job :)

>
> I should be able to post an updated version of the
> script with what you asked tomorrow, then I'll
> start writing the modules to install the
> recovery.
>
> bodhi Wrote:
> --------------------------------------------------
> -----
> >
> > I'm thinking about newer boxes that we don't
> have
> > support yet. They use kernel 3.2.x. So yes I
> think
> > "rootfs" is what we really care about.
> >
>
> As I said there is no reliable way of
> autodetecting the device of / so the script has no
> way to know if the rootfs is actually / (an issue)
> or if it is not / (not an issue).

Stock might not have it. But if the system is running is more modern Linux then there should be way, right? such as

ll /dev/disk/by-label/
total 0
drwxr-xr-x 2 root root  60 Jul 18 16:56 .
drwxr-xr-x 5 root root 100 Jul 18 16:56 ..
lrwxrwxrwx 1 root root  10 Jul 18 16:56 rootfs -> ../../sda1

What Ray said above is good for Debian, but if the user is installing on a running Arch, Fedora, Gentoo, or whatever else then we'll have to reply on standard Linux devices in /dev. So if rootfs is the label of some disk, and also /etc/ is on it then the running rootfs is on that.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Updated the script with feedback.

Now there are two error message functions, that print boilerplate text around a specific error line, so I could wire all errors firing around the script to one of them.
Now each error that causes a script termination has a big notice with further instructions for user.
I still need to add the new errors to the partitioning and extracting rootfs part.

I added the filter to get "orion_nand" in mtdparts.

I added a basic bad blocks check, which does this

for f in 20000 40000 60000 80000 a0000 c0000 e0000 100000 ; do

# call dmesg, show only lines with "bad" lower or uppercase,
#remove "0x" and any trailing zeroes so we get only the actual number of the hex address
#then finally check if in these lines there is the address we are looking for (both lower or upper and single word
#so we don't match also 200000 when looking for 20000)
dmesg | grep -i 'bad' | sed 's/0x[0-0][0-0]*//' | grep -iw $f

if [ "$?" = "0" ]; then
    ERROR="## Bad blocks in 0x$f area on NAND, uboot must be installed manually."
    error_harmless "$ERROR"
    exit 1
fi

done

It should basically look in dmesg recursively and if it finds a line stating "bad/Bad/BAD blablablablablabla HEX_ADDRESS" it will trigger an error function to write the error and quit the script.

this is an example output of dmesg

# dmesg | grep -i 'bad'
#
# [    2.413231] Scanning device for bad blocks
# [    2.417731] Bad eraseblock 3 at 0x000000060000
#

This function should also allow easy modifications in the future if I need to check if there are more than X bad blocks or if the bad block is exactly in c0000 where we should place envs, as it is checking each and every address anyway.

I didn't add anything to check about the rootfs matter as I need more time and more tests about that.


Now the script should be good enough to start adding device support to it, which is why I'm uploading it now on this post.



Edited 7 time(s). Last edit at 09/11/2016 01:53AM by bobafetthotmail.

Thanks. I'll start with the Pogo V4. I think I can boot back to stock with this box. And the next will be Pogo E02.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

I am testing the Pogo V4 installation.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

bobafetthotmail,

While adding Pogo V4, I've discovered some problem with the bad blocks checking. So what I've modified in this attached version 14:

1. Add Pogo V4
2. Fill in pogoplug_kill_programs

3. Fix the bad blocks checking (2 parts). The list of addresses and the "Bad eraseblock".

a. The list of addresses should be

for f in 00000 20000 40000 60000 80000 a0000 c0000 e0000 ; do

b. Interesting that the check for bad blocks that we always do manually is not enough in a script! In stock, I got this extremely rare case: the time tag contains "40000" and the same line is about "bad PEB" :)) So when the installer saw that, it aborted. I've fixed this by using the string "Bad eraseblock".

/ # dmesg | grep -i bad
<6>[    1.010000] Scanning device for bad blocks
<5>[    1.370000] UBI: number of bad PEBs:         0
<5>[    1.400000] UBI: number of PEBs reserved for bad PEB handling: 8

Please do a diff to your version to see if I got this correction right. Also, I used OSX Xcode to modify this script, so not sure whether it did replace some tabs space. Let me know if you have any problem with the diff.


I'm still testing this. Will continue tomorrow.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 09/24/2016 05:47AM by bodhi.

Ah, no hard bricking, very good news. :)

From a look at the parts you added/modified it should be OK. I didn't check the correctness of the end-of-line encoding because OSX is a Unix system so it does use the same encoding as Linux by default.


I'm nearly ready to release version 14 myself, where the script can flash the recovery too, I'll merge your fixes and device support when I have finished.

I was planning to upload the final version to github in your repo, so you/me/everyone could add fixes/support without asking me in the future.

bobafetthotmail Wrote:
-------------------------------------------------------
> Ah, no hard bricking, very good news. :)
>
> From a look at the parts you added/modified it
> should be OK. I didn't check the correctness of
> the end-of-line encoding because OSX is a Unix
> system so it does use the same encoding as Linux
> by default.
>

True. I never had problem using Xcode to edit Linux scripts (it is a nice tool, albeit way overkilled using as an editor :) But when I did a diff this time, it shown the differences were more than I added, and it seems some white spaces are different.


>
> I'm nearly ready to release version 14 myself,
> where the script can flash the recovery too, I'll
> merge your fixes and device support when I have
> finished.
>
> I was planning to upload the final version to
> github in your repo, so you/me/everyone could add
> fixes/support without asking me in the future.

Cool!

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

One more thing.

I don't like to pull wget from Amazon AWS. This wget should be pulled from the same place as the other tool binaries. So I planned to recompile it. But I never seem to have time to look for my old squeeze rootfs :)

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 09/24/2016 03:38PM by bodhi.

I have some problem with u-boot envs. But it coud have been because my current stock OS envs were corrupted. I'll have to flash NAND mtds back to stock and try again.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

1. Any idea why these certificate checking failed? ah, looks like it was the URL differences.

Connecting to pogoplug.s3.amazonaws.com (54.231.168.230:80)
--2016-09-24 13:29:59--  https://github.com/mibodhi/kirkwood/blob/kirkwood-2016-r1/tools/busybox?raw=true
Resolving github.com... 192.30.253.113
Connecting to github.com|192.30.253.113|:443... connected.
WARNING: cannot verify github.com's certificate, issued by `/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 Extended Validation Server CA':
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 302 Found
Location: https://github.com/mibodhi/kirkwood/raw/kirkwood-2016-r1/tools/busybox [following]
--2016-09-24 13:29:59--  https://github.com/mibodhi/kirkwood/raw/kirkwood-2016-r1/tools/busybox
Connecting to github.com|192.30.253.113|:443... connected.
WARNING: cannot verify github.com's certificate, issued by `/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 Extended Validation Server CA':
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/mibodhi/kirkwood/kirkwood-2016-r1/tools/busybox [following]
--2016-09-24 13:30:00--  https://raw.githubusercontent.com/mibodhi/kirkwood/kirkwood-2016-r1/tools/busybox
Resolving raw.githubusercontent.com... 151.101.24.133
Connecting to raw.githubusercontent.com|151.101.24.133|:443... connected.
WARNING: cannot verify raw.githubusercontent.com's certificate, issued by `/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA':
  Unable to locally verify the issuer's authority.
WARNING: certificate common name `www.github.com' doesn't match requested host name `raw.githubusercontent.com'.
HTTP request sent, awaiting response... 200 OK
Length: 1616492 (1.5M) [application/octet-stream]

2. Perhaps we want to suppress the tar command output. The entire rootfs files listing are printed out here!

Saving to: `/tmp/rootfs/debian-kirkwood-rootfs-bodhi-latest.tar.bz2'

100%[======================================>] 143,314,034  312K/s   in 7m 48s  

2016-09-24 13:44:07 (299 KB/s) - `/tmp/rootfs/debian-kirkwood-rootfs-bodhi-latest.tar.bz2' saved [143314034/143314034]

verify file /tmp/rootfs/debian-kirkwood-rootfs-bodhi-latest.tar.bz2 with md5 /tmp/rootfs/debian-kirkwood-rootfs-bodhi-latest.tar.bz2.md5
9f957e1dc5a36f90a285ab4615cb02fd
9f957e1dc5a36f90a285ab4615cb02fd
passed
---extracting debian rootfs to install Debian
./
./proc/
./sbin/
./sbin/poweroff
./sbin/ldconfig
./sbin/capsh
./sbin/dhclient-script
./sbin/rpcbind
./sbin/rpc.statd
./sbin/showmount
./sbin/fsck.minix
./

....

3. I think something went wrong here. It was not because my stock envs. But these errors seem to be path-related.

partition to mount is /dev/sda1
trying to read the env with stock blparam
./kirkwood-installer: line 2308: blparam: not found
if there is fw_prinenv in stock fw we use it
./kirkwood-installer: line 2308: fw_printenv: not found
we try with our own fw_printenv, hoping there is a config in the right place
Cannot parse config file: No such file or directory
the above command failed again, we return an error code
we now try reading the envs from an older version of our own uboot envs
Warning: Bad CRC, using default environment
## Error: "" not defined
reading stock_env_backup.txt
cat: can't open '/tmp/rootfs/uboot_backups/stock_env_backup2016-09-24_13:55.txt': No such file or directory
reading modern_env_backup.txt
cat: can't open '/tmp/rootfs/uboot_backups/modern_env_backup2016-09-24_13:55.txt': No such file or directory
Block size 131072, page size 2048, OOB size 64
Dumping data starting at 0x00000000 and ending at 0x00080000...
trying to read the env with stock blparam
./kirkwood-installer: line 2308: blparam: not found
we try reading envs by using stock fw_printenv
./kirkwood-installer: line 2308: fw_printenv: not found
we try with our own fw_printenv, hoping there is a config in the right place
./kirkwood-installer: line 2308: fw_printenv: not found
the above command failed again, we return an error code
we now try reading the envs from an older version of our own uboot envs
Warning: Bad CRC, using default environment
## Error: "ethaddr" not defined
sh: : unknown operand
trying to read the env with stock blparam
./kirkwood-installer: line 2308: blparam: not found
we try reading envs by using stock fw_printenv
./kirkwood-installer: line 2308: fw_printenv: not found
we try with our own fw_printenv, hoping there is a config in the right place
./kirkwood-installer: line 2308: fw_printenv: not found
the above command failed again, we return an error code
we now try reading the envs from an older version of our own uboot envs
Warning: Bad CRC, using default environment
## Error: "eth1addr" not defined
sh: : unknown operand
trying to read the env with stock blparam
./kirkwood-installer: line 2308: blparam: not found
we try reading envs by using stock fw_printenv
./kirkwood-installer: line 2308: fw_printenv: not found
we try with our own fw_printenv, hoping there is a config in the right place
./kirkwood-installer: line 2308: fw_printenv: not found
the above command failed again, we return an error code
trying to read the env with stock blparam
./kirkwood-installer: line 2308: blparam: not found
we try reading envs by using stock fw_printenv
./kirkwood-installer: line 2308: fw_printenv: not found
we try with our own fw_printenv, hoping there is a config in the right place
./kirkwood-installer: line 2308: fw_printenv: not found
the above command failed again, we return an error code
read bootargs as 
MTDPARTS is 
we now try reading the envs from an older version of our own uboot envs
Warning: Bad CRC, using default environment
## Error: "mtdparts" not defined
sh: : unknown operand
MTDPARTS is 
We are now ready to flash uboot environment, its boot configuration.

If this is the first time you run this script, answer Y

If you are updating a previous version of bodhi's uboot and have
custom uboot envs you wish to keep, answer N

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

FYI, regarding number 3 above

Quote

3. I think something went wrong here. It was not because my stock envs. But these errors seem to be path-related.

The stock blparam is in /usr/local/cloudengines/bin/blparam. And this cloudengines directory is not in stock OS bin PATH. That's why the script could not find it.

echo "trying to read the env with stock blparam"
  VALUE=$(blparam | grep "^$ENV_NAME=" | sed "s@$ENV_NAME=@@g"  )

perhaps just do this

echo "trying to read the env with stock blparam"
  VALUE=$(/usr/local/cloudengines/bin/blparam | grep "^$ENV_NAME=" | sed "s@$ENV_NAME=@@g"  )

Regardless, the fall back is using our own blparam, but perhaps the PATH was not restored back to $TOOL properly at some point.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 09/24/2016 06:53PM by bodhi.

bodhi Wrote:
-------------------------------------------------------

>I don't like to pull wget from Amazon AWS. This wget should be pulled from the same place as the other tool binaries. So I planned to recompile it. But I never seem to have time to look for my old squeeze rootfs :)

yeah it's better to have it on something we can control. Just remember that the wget is on Amazon AWS as it MUST stay on a plain ftp or http link as we are using stock wget to download it, so it cannot go on github and probably must stay on this site.

There are also a few other tools (blparam and mke2fs, gdisk/sgdisk don't work so let them stay there) that still come from my dropbox that should come from github instead, and also the md5 of tools still come from my dropbox.

I don't see why we should recompile wget with ssl, the current binary works fine, if you want to recompile something statically I still need a sgdisk to make gpt partitions.

> 1. Any idea why these certificate checking failed?
> ah, looks like it was the URL differences.
>

stock firmware has no certificate storage so even if wget is compiled with ssl and it can download from https it cannot check if the certificates are valid or not.
the wget is getting called with --ignore-certificates (or something) so that is just a warning, if I didn't then it would be an error and the download would abort.
I think it's possible to get also the certificate package in there to do proper certificate checking but I think it's overkill.

>
> 2. Perhaps we want to suppress the tar command
> output. The entire rootfs files listing are
> printed out here!

I prefer to let the script be talkative than idle for a long time on a blank line.
If someone is looking at the console he will see that things are happening, and will not be worried that something broke or did hang.

>
> 3. I think something went wrong here. It was not
> because my stock envs. But these errors seem to be
> path-related.

currently the script isn't using our own blparam, it tries blparam from stock PATH, then tries stock fw_printenv, then tries with our own printenv and stock config, then with our fw_printenv and our own config.

And from the logs it seems it isn't finding stock blparam. Atm I don't recall exactly why I did try only with stock and not with ours, I think it was because I was not sure ours would work on all devices that used blparam.

>The stock blparam is in /usr/local/cloudengines/bin/blparam. And this cloudengines directory is not in stock OS bin PATH. That's why the script could not find it.

yeah that would explain it. I guess it wasn't included in PATH for safety reasons or whatever.

>Regardless, the fall back is using our own blparam,

is our own blparam going to read envs for all devices using blparam (pogoplugs and goflex boxes)? because if it works with all such devices I'm going to change the "we try with stock blparam" code into a "we use our own blparam regardless" instead, no try with stock at all.

EDIT: uploading a testing script for my own needs, don't use plz.



Edited 1 time(s). Last edit at 09/25/2016 06:03AM by bobafetthotmail.

bobafetthotmail Wrote:

>
> yeah it's better to have it on something we can
> control. Just remember that the wget is on Amazon
> AWS as it MUST stay on a plain ftp or http link as
> we are using stock wget to download it, so it
> cannot go on github and probably must stay on this
> site.
>
> There are also a few other tools (blparam and
> mke2fs, gdisk/sgdisk don't work so let them stay
> there) that still come from my dropbox that should
> come from github instead, and also the md5 of
> tools still come from my dropbox.
>
> I don't see why we should recompile wget with ssl,
> the current binary works fine,

Agreed. I'll copy wget to the same place as ours bin box.

> the wget is getting called with
> --ignore-certificates (or something) so that is
> just a warning, if I didn't then it would be an
> error and the download would abort.
> I think it's possible to get also the certificate
> package in there to do proper certificate checking
> but I think it's overkill.

OK.

> I prefer to let the script be talkative than idle
> for a long time on a blank line.
> If someone is looking at the console he will see
> that things are happening, and will not be worried
> that something broke or did hang.

OK. It makes sense.

> And from the logs it seems it isn't finding stock
> blparam. Atm I don't recall exactly why I did try
> only with stock and not with ours, I think it was
> because I was not sure ours would work on all
> devices that used blparam.
>
> >The stock blparam is in
> /usr/local/cloudengines/bin/blparam. And this
> cloudengines directory is not in stock OS bin
> PATH. That's why the script could not find it.
>
> yeah that would explain it. I guess it wasn't
> included in PATH for safety reasons or whatever.
>

Yes. They don't normal users to mess up NAND. And blparam output looks scary, too! usually, I recommend people to redirect output to /dev/null when changing it, because they were panic when seeing the output :) Listing envs is fine, it looks normal.

> is our own blparam going to read envs for all
> devices using blparam (pogoplugs and goflex
> boxes)? because if it works with all such devices
> I'm going to change the "we try with stock
> blparam" code into a "we use our own blparam
> regardless" instead, no try with stock at all.

As I recall it does work. They all use the same stock OS. But let's try their path for stock first, and fallback to ours.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

hm, btw, I'll also have to look at why the checks that abort the installation if the envs can't be read didn't work.
Because it was supposed to abort if it could not read crucial envs like mac address and mtdparts.

bobafetthotmail Wrote:
-------------------------------------------------------
> hm, btw, I'll also have to look at why the checks
> that abort the installation if the envs can't be
> read didn't work.
> Because it was supposed to abort if it could not
> read crucial envs like mac address and mtdparts.

That's true.

I did complete the installation, I was able to ignore the errors about those envs not found. At the end, the default envs were the deafults: pogo_e02 DTB (dtb_file), and mtdparts for other Kirwood boxes (mtdparts=mtdparts=orion_nand:1M(u-boot),4M(uImage),32M(rootfs),-(data)).

So I monitored serial console, interrupted it, and verified. It looks like the envs are all default envs.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 09/24/2016 10:56PM by bodhi.

I've fixed the issue above, and now if crucial envs cannot be read it will stop.

I'm having issues with checking that the recovery was flashed correctly. I'm making a nanddump with the file lenght, but I'm getting a bigger file
this is the line

/tmp/tools/nanddump --noecc --omitoob -l 0x1CA8A4 -f /tmp/tools/recovery-uImage.dump /dev/mtd7

the hex after the -l is the file size of the file I flashed in hexadecimal, or 1878180 bytes in decimal.

If i check the size of the file I dumped with wc -c I get that it is 1880064 bytes long

I'm suspecting a padding issue as the file I flash isn't pre-padded by the lede build system (the rootfs image is).

How can I get nanddump to give me a file the right size for md5 checking? Do I need to pad the starting file? How?


EDIT: I solved this on my own.
Yes I needed to pad it to the closest multiple of 128k bytes (block size), I wrote truncate -s 1880064 <file-name-to-pad> to pad it and updated the checksum, now it flashes correctly.



Edited 1 time(s). Last edit at 09/26/2016 08:59AM by bobafetthotmail.

bobafetthotmail Wrote:
-------------------------------------------------------
> I've fixed the issue above, and now if crucial
> envs cannot be read it will stop.
>
> I'm having issues with checking that the recovery
> was flashed correctly. I'm making a nanddump with
> the file lenght, but I'm getting a bigger file
> this is the line
>

> /tmp/tools/nanddump --noecc --omitoob -l 0x1CA8A4
> -f /tmp/tools/recovery-uImage.dump /dev/mtd7
>

> the hex after the -l is the file size of the file
> I flashed in hexadecimal, or 1878180 bytes in
> decimal.
>
> If i check the size of the file I dumped with
> wc -c I get that it is 1880064 bytes long
>
> I'm suspecting a padding issue as the file I flash
> isn't pre-padded by the lede build system (the
> rootfs image is).
>
> How can I get nanddump to give me a file the right
> size for md5 checking? Do I need to pad the
> starting file? How?
>
>
> EDIT: I solved this on my own.
> Yes I needed to pad it to the closest multiple of
> 128k bytes (block size), I wrote truncate -s
> 1880064 <file-name-to-pad> to pad it and
> updated the checksum, now it flashes correctly.

Actually the normal way to do this is to make the image multiple of 128K using dd. So all u-boot images that I've released are 512K. This should be applicable to all NAND flash images. For NOR flash, pad images to multiple of 64K.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)