Help needed to improve proposed migration

Hi!

Every once in a while, in the Foundations team, we do a coding day. A year ago, Lukasz and I wrote a script, following an idea from Steve Langasek, to provide "hints" and help for the next steps necessary for a package to migrate from -proposed to -release.

"ubuntu-archive-assistant" was born. I just pushed this to lp:ubuntu-dev-tools, after it being on its own in a separate git tree for a long while. I'd love to get help for feedback, as well as more people contributing fixes, etc. ubuntu-archive-assistant is designed to let you look at a specific package in -proposed and try to tell you what to do next to ensure it migrates from -proposed.

This is great work for new contributors wishing to work on something, say, to get upload privileges in Ubuntu.

Here's how it works (it uses subcommands right now).

Without any further options than "ubuntu-archive-assistant proposed", it will list packages in -proposed and let you pick:

$ ./ubuntu-archive-assistant proposed
No source package name was provided. The following packages are blocked in proposed:

(1) gnome-shell-extension-multi-monitors (Age: 338 days)
(2) node-is-glob (Age: 278 days)
(3) node-concat-with-sourcemaps (Age: 264 days)
(4) node-postcss (Age: 231 days)
(5) node-source-map (Age: 229 days)
(6) android-platform-system-core (Age: 226 days)
(7) libdigidocpp (Age: 226 days)
(8) qesteidutil (Age: 225 days)
(9) schleuder (Age: 218 days)
(10) ncbi-blast+ (Age: 216 days)
(11) node-postcss-filter-plugins (Age: 213 days)
(12) node-postcss-load-options (Age: 213 days)
(13) node-postcss-load-plugins (Age: 213 days)
(14) node-postcss-minify-font-values (Age: 213 days)
(15) node-postcss-load-config (Age: 209 days)
(16) live-config (Age: 207 days)
Page -1-. Press any key for next page or Q to select a package.
Which package do you want to look at? 9
Next steps for schleuder 3.2.2-1:
  Fix missing builds: amd64
     https://launchpad.net/ubuntu/+source/schleuder/3.2.2-1

If you specify which package you want to look at, it will give you the specifics for that package (examples here are for what is currently in -proposed):

$ ./ubuntu-archive-assistant proposed -s qesteidutil
Next steps for qesteidutil 0.3.1-0ubuntu4:
  Fix missing builds: amd64, arm64, armhf, i386, ppc64el, s390x
     https://launchpad.net/ubuntu/+source/qesteidutil/0.3.1-0ubuntu4

$ ./ubuntu-archive-assistant proposed -s android-platform-system-core
Next steps for android-platform-system-core 1:7.0.0+r33-2build1:
  Fix missing builds: amd64, arm64, armhf, i386
     https://launchpad.net/ubuntu/+source/android-platform-system-core/1:7.0.0+r33-2build1 

You can even get more information about the next steps for a package, by enabling --debug or --verbose:

$ ./ubuntu-archive-assistant proposed -s live-config                 Next steps for live-config 5.20180224:
  Fix unsatisfiable dependencies in live-config:

$ ./ubuntu-archive-assistant proposed --verbose -s live-config
live-config is not considered ✘
Next steps for live-config 5.20180224:
  Fix unsatisfiable dependencies in live-config:
    sysvinit-core | sysvinit (<< 2.88dsf-44) can not be satisfied on amd64 ✘
      sysvinit-core only exists in Debian ✘
 

$ ./ubuntu-archive-assistant proposed --debug -s live-config
live-config is not considered ✘
Next steps for live-config 5.20180224:
DEBUG: reasons: ['depends'] 
  Fix unsatisfiable dependencies in live-config:
    sysvinit-core | sysvinit (<< 2.88dsf-44) can not be satisfied on amd64 ✘
      sysvinit-core only exists in Debian ✘
         DEBUG: Is this package blacklisted? Should it be synced?

We've covered some of the common reasons for a package to be stuck in proposed, but there are a ton of others. We'll need help to improve the tooling and make it useful for everyone wishing to work on proposed migration. There's a lot more that can be done, including spending time to parse update_output.txt (or better yet, a YAML representation of it) and testing package installation automatically to figure out what packages need no-change rebuilds, etc. A lot of it is integration of other tools that already exist.

That's where you come in.

This is a great way to learn a lot more about what happens to packages after they are uploaded, and what more you can do to ensure your own uploads move quickly to be accessible to all Ubuntu users; and many of the improvements can be as simple as contributing a simple test for one failure case for packages in -proposed.

More to come about ubuntu-archive-assistant. There are other subcommands in it than just "proposed". :)

Backing up GPG keys

Using PGP/GPG keys for a long period of time (either expiring keys, or extending expiration dates) and the potential for travel, for hardware to fail, or for life's other events means that eventually rather than potentially, you will end up in a situation where a key is lost, damaged, or where you otherwise need to proceed with some disaster recovery techniques.

These techniques could be as simple as forgetting about the key altogether and letting it live forever on the Internet, without being used. It could also be that you were clever and saved a revocation certificate somewhere different than your private key is backed up, but what if you didn't?

What if you did not print the revocation certificate? Or you just really don't feel very much like re-typing half a gazillion character?

I wouldn't wish it to anyone, but there will always be the risk of a failure of your "backup options"; so I'm sharing here my personal backup methods.

I back up my GPG keys, which I use both at and outside of work, on multiple different media:

• "Daily use" happens using a Yubikey that holds securely the private part of the keys (it can't be extracted from the smartcard), as well as the public part. I've already written about this two years ago, on this blog.
• The first layer of backup is on a LUKS-encrypted USB key. The USB key must obviously be encrypted to block out most attempts at accessing the contents of the key; and it is a key that I usually carry on my person at all times, like the Yubikeys -- I also use it to back up other files I can't leave without, such as a password vault, some other certificates, copies of ID documents in case of loss for when I travel, etc.
• The next layer is on paper. Well, cardstock actually, to avoid wanting to fold it. This is the process I want to dig into deeper here.

It turns out that backing up secure keys on paper is pretty straightforward, and something just fine to do. You will obviously want to keep the paper copies in a secure location that only you have access to, as much as possible safe from fire (or at least somewhere unlikely to burn down at the same time as you'd lose the other backups).

paperkey is a generally accepted way of saving the private part of your GPG key. It does a decent job at saving things in a printable form, from which point you would go ahead and re-type, or use OCR to recover the text generated by paperkey:

paperkey --secret-key secret.gpg --output printme.txt

This retains the same security systems as your original key. You should have added a passphrase to it anyway, so even if the paper copy was found and used to recover the key, you would be protected by the complexity of your passphrase.

But this depends on OCR working correctly, especially on an aging medium such as paper, or you spending many hours re-typing the contents, and potentially tracking down typos. There's error correction, but that sounds to me like not fun at all. When you want to recover your key, presumably it is because you really do need it as soon as possible.

Back in 2015 when I generated my latest keys, I found a blog post that explained how to use QR codes to back up data. QR codes have the benefit of being very resilient to corruption, and above all, do not require typing. QR codes are however limited in size, being limited to 177x177 squares, for about 1200 characters storage.

Along with that blog post, I also found out about DataMatrix codes (which are quite similar to QR codes), but where each symbol can save a bit more data (about 1500 bytes per image in the biggest size). Pick the format you prefer, I picked DataMatrix. Simply modify the size you split to in the commands below.

One might wish to save the paperkey or the private key directly (obviously, saving the private key might mean more chunks to print), and that can be done using the programs in dmtx-utils:

cat printme.txt | split -b 1500 - part-
rm printme.txt
for part in part-*; do
    dmtxwrite -e 8 ${part} > ${part}.png
done 

You will be left with multiple parts of the file you originally split (without a file extension), as well as a corresponding image in PNG format that can be printed, and later scanned, to recover the original.

Keep these in a safe location and your key should be recoverable years down the line. It's not a bad idea to "pretend" there's a catastrophe and attempt to recover your key every few months, just to be sure you can go through the steps easily and that the paper keys are in good shape.

Recovery is simple:

for file in *.png; do dmtxread $file >> printme.txt; done

If all went well, the original and recovered files should be identical, and you just avoided a couple of hours of typing.

Stay safe!

ss: another way to get socket statistics

In my last blog post I mentioned ss, another tool that comes with the iproute2 package and allows you to query statistics about sockets. The same thing that can be done with netstat, with the added benefit that it is typically a little bit faster, and shorter to type.

Just ss by default will display much the same thing as netstat, and can be similarly passed options to limit the output to just what you want. For instance:

$ ss -t
State       Recv-Q Send-Q       Local Address:Port                        Peer Address:Port              
ESTAB       0      0                127.0.0.1:postgresql                     127.0.0.1:48154              
ESTAB       0      0            192.168.0.136:35296                      192.168.0.120:8009                
ESTAB       0      0            192.168.0.136:47574                     173.194.74.189:https

[...]

ss -t shows just TCP connections. ss -u can be used to show UDP connections, -l will show only listening ports, and things can be further filtered to just the information you want.

I have not tested all the possible options, but you can even forcibly close sockets with -K.

One place where ss really shines though is in its filtering capabilities. Let's list all connections with a source port of 22 (ssh):

$ ss state all sport = :ssh
Netid State      Recv-Q Send-Q     Local Address:Port                      Peer Address:Port              
tcp   LISTEN     0      128                    *:ssh                                  *:*                  
tcp   ESTAB      0      0          192.168.0.136:ssh                      192.168.0.102:46540              
tcp   LISTEN     0      128                   :::ssh                                 :::* 

And if I want to show only connected sockets (everything but listening or closed):

$ ss state connected sport = :ssh
Netid State      Recv-Q Send-Q     Local Address:Port                      Peer Address:Port              
tcp   ESTAB      0      0          192.168.0.136:ssh                      192.168.0.102:46540 

Similarly, you can have it list all connections to a specific host or range; in this case, using the 74.125.0.0/16 subnet, which apparently belongs to Google:

$ ss state all dst 74.125.0.0/16
Netid State      Recv-Q Send-Q     Local Address:Port                      Peer Address:Port              
tcp   ESTAB      0      0          192.168.0.136:33616                   74.125.142.189:https              
tcp   ESTAB      0      0          192.168.0.136:42034                    74.125.70.189:https              
tcp   ESTAB      0      0          192.168.0.136:57408                   74.125.202.189:https        

This is very much the same syntax as for iptables, so if you're familiar with that already, it will be quite easy to pick up. You can also install the iproute2-doc package, and look in /usr/share/doc/iproute2-doc/ss.html for the full documentation.

Try it for yourself! You'll see how well it works. If anything, I'm glad for the fewer characters this makes me type.

If you're still using ifconfig, you're living in the past

The world evolves

I regularly see "recommendations" to use ifconfig to get interface information in mailing list posts or bug reports and other places. I might even be guilty of it myself. Still, the world of networking has evolved quite a lot since ifconfig was the de-facto standard to bring up a device, check its IP or set an IP.

Following some improvements in the kernel and the gradual move to driving network things via netlink; ifconfig has been largely replaced by the ip command.

Running just ip yields the following:

Usage: ip [ OPTIONS ] OBJECT { COMMAND | help }
       ip [ -force ] -batch filename
where  OBJECT := { link | address | addrlabel | route | rule | neigh | ntable |
                   tunnel | tuntap | maddress | mroute | mrule | monitor | xfrm |
                   netns | l2tp | fou | macsec | tcp_metrics | token | netconf | ila }
       OPTIONS := { -V[ersion] | -s[tatistics] | -d[etails] | -r[esolve] |
                    -h[uman-readable] | -iec |
                    -f[amily] { inet | inet6 | ipx | dnet | mpls | bridge | link } |
                    -4 | -6 | -I | -D | -B | -0 |
                    -l[oops] { maximum-addr-flush-attempts } | -br[ief] |
                    -o[neline] | -t[imestamp] | -ts[hort] | -b[atch] [filename] |
                    -rc[vbuf] [size] | -n[etns] name | -a[ll] | -c[olor]}

I understand this may look complicated to some people, but the jist of it is to understand that with ip, you interact with objects, and apply some kind of function to it. For example:

ip address show

This is the main command that would be used in place of ifconfig. It will just display the IP addresses assigned to all interfaces. To be precise, it will show you the layer 3 details the interface: the IPv4 and IPv6 addresses, whether it is up, what are the different properties related to the addresses...

Another command will give you details about the layer 2 properties of the interface: its MAC address (ethernet address), etc; even if it is shown by ip address:

ip link show

Furthermore, you can set devices up or down (similar to ifconfig eth0 up or ifconfig eth0 down) simply by using:

ip link set DEVICE up or ip link set DEVICE down

As shown above, there are lots of other objects that can be interacted with using the ip command. I'll cover another: ip route, in another post.

Why is this important?

As time passes, more and more features are becoming easier to use with the ip command instead of with ifconfig. We've already stopped installing ifconfig on desktops (it still gets installed on servers for now), and people have been discussing dropping net-tools (the package that ships ifconfig and a few other old commands that are replaced) for a while now. It may be time to revisit not installing net-tools by default anywhere.

I want to know about your world

Are you still using one of the following tools?

/bin/netstat    (replaced by ss, for which I'll dedicate another blog post entirely)

/sbin/ifconfig

/sbin/ipmaddr   (replaced by ip maddress)

/sbin/iptunnel

/sbin/mii-tool    (ethtool should appropriately replace it)

/sbin/nameif

/sbin/plipconfig

/sbin/rarp

/sbin/route

/sbin/slattach

If so and there is just no alternative to using them that comes from iproute2 (well, the ip or ss commands) that you can use to do the same, I want to know about how you are using them. We're always watching for things that might be broken by changes; we want to avoid breaking things when possible.

In full tinfoil hat mode: Using GPG with smartcards

Breaking OPSEC for a bit to write a how-to on using GPG keys with smartcards...

I've thought about experimenting with smartcards for a while. Turns out that my Thinkpad has a built-in smartcard reader, but most of my other systems don't. Also, I'd like to use a smartcard to protect my SSH keys, some of which I may use on systems that I do not fully control (ie. at the university to push code to Github or Bitbucket), or to get to my server. Smartcard readers are great, but they're not much fun to add to a list of stuff to carry everywhere.

There's an alternate option: the Yubikey. Yubico appears to have made a version 4 of the Yubikey which has CCID (smartcard magic), U2F (2-factor for GitHub and Google, on Chrome), and their usual OTP token, all on the same tiny USB key. What's more, it is documented as supporting 4096 bit RSA keys, and includes some ECC support (more on this later).

Setting up GPG keys for use with smartcards is simple. One has the choice of either creating your own keys locally, and moving them on the smartcard, or generating them on the smartcard right away. In other to have a backup of my full key available in a secure location, I've opted to generate the keys off of the card, and transferring them.

For this, you will need one (or two) Yubikey 4 (or Yubikey 4 Nano, or if you don't mind being limited to 2048 bit keys, the Yubikey NEO, which can also do NFC), some backup media of your choice, and apparently, at least the following packages:

gnupg2 gnupg-agent libpth20 libccid pcscd scdaemon libksba8 opensc

You should do all of this on a trusted system, not connected to any network.

First, setup gnupg2 to a reasonable level of security. Edit ~/.gnupg/gpg.conf to pick the options you want, I've based my config on Jeffrey Clement's blog entry on the subject:

#default-key AABBCC90DEADBEEF
keyserver hkp://keyserver.ubuntu.com
no-emit-version
no-comments
keyid-format 0xlong
with-fingerprint
use-agent
personal-cipher-preferences AES256 AES192 AES CAST5
personal-digest-preferences SHA512 SHA384 SHA256 SHA224
cert-digest-algo SHA512
default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed

You'll want to replace default-key later with the key you've created, and uncomment the line.

The downside to all of this is that you'll need to use gpg2 in all cases rather than gpg; which is still the default on Ubuntu and Debian. gpg2 so far seems to work just fine for ever use I've had (including debsign, after setting DEBSIGN_PROGRAM=gpg2 in ~/.devscripts).

You can now generate your master key:

gpg2 --gen-key

Then edit the key to add new UIDs (identities) and subkeys, which will each have their own different capabilities:

gpg2 --expert --edit-key 0xAABBCC90DEADBEEF

Best is to follow jclement's blog entry for this. There is no point in reiterating all of it. There's also a pretty complete guide from The Linux Foundation IT here, though it seems to include a lot of stuff that does not appear to be required here on my system, in xenial.

Add the subkeys. You should have one of encryption, one for signing, and one for authentication. Works out pretty well, since there are three slots, one for each of these capabilities, on the Yubikey.

If you also want your master key on a smartcard, you'll probably need a second Yubikey (that's why I wrote two earlier), which would only get used to sign other people's keys, extend expiration dates, generate new subkeys, etc. That one should be left in a very secure location.

This is a great point to backup all the keys you've just created:

gpg2 -a --export-secret-keys 0xAABBCC90DEADBEEF > 0xAABBCC90DEADBEEF.master.key
gpg2 -a --export-secret-subkeys 0xAABBCC90DEADBEEF > 0xAABBCC90DEADBEEF.sub.key
gpg2 -a --export 0xAABBCC90DEADBEEF > 0xAABBCC90DEADBEEF.pub

Next step is to configure the smartcard/Yubikey to add your name, a URL for the public key, set the PINs, etc. Use the following command for this:

gpg2 --card-edit

Finally, go back to editing your GPG key:

gpg2 --expert --edit-key 0xAABBCC90DEADBEEF

From this point you can use toggle to select each subkey (using key #), move them to the smartcard (keytocard), and deselect them (key #). To move the master key to the card, "toggle" out of toggle mode then back in, then immediately run 'keytocard'. GPG will ask if you're certain. There is no way to get a key back out of the card, if you want a local copy, you needed to make a backup first.

Now's probably a great time to copy your key to a keyserver, so that people may eventually start to use it to send you encrypted mail, etc.

After transferring the keys, you may want to make a "second backup", which would only contain the "clues" for GPG to know on which smartcard to find the private part of your keys. This will be useful if you need to use the keys on another system.

Another option is to use the public portion of your key (saved somewhere, like on a keyserver), then have gpg2 discover that it's on a smartcard using:

gpg2 --card-status

Unfortunately, it appears to only manage to pick up either only the master key, or only the subkeys, if you use separate smartcards. This may be a blessing in disguise, in that you'd still only use the master key on an offline, very secure system, and only the subkeys in your typical daily use scenario.

Don't forget to generate a revocation certificate. This is essential if you ever lose your key, if it's compromised, or you're ever in a situation where you want to let the world know quickly not to use your key anymore:

gpg2 --gen-revoke 0xAABBCC90DEADBEEF

Store that data in a safe place.

Finally, more on backing up the GPG keys. It could be argued that keeping your master key on a smartcard might be a bad idea. After all, if the smartcard is lost, while it would be difficult to get the key out of the smartcard, you would probably want to treat it as compromised and get the key revoked. The same applies to keys kept on USB drives or on CD. A strong passphrase will help, but you still lost control of your key and at that point, no longer know whether it is still safe.

What's more, USB drives and CDs tend to eventually fail. CDs rot after a number of years, and USB drives just seem to not want to work correctly when you really need them. Paper is another option for backing up your keys, since there are ways (paperkey, for instance) to represent the data in a way that it could either be retyped or scanned back into digital data to be retrieved. Further securing a backup key could involve using gfshare to split it into multiple bits, in the hope that while one of its locations could be compromised (lost), you'll still have some of the others sufficient to reconstruct the key.

With the subkeys on the Yubikey, and provided gpg2 --card-status reports your card as detected, if you have the gpg-agent running with SSH support enabled you should be able to just run:

ssh-add -l

And have it list your card serial number. You can then use ssh-add -L to get the public key to use to add to authorized_keys files to use your authentication GPG subkey as a SSH key. If it doesn't work, make sure the gpg-agent is running and that ssh-add uses the right socket, and make sure pcscd isn't interfering (it seemed to get stuck in a weird state, and not shutting down automatically as it should after dealing with a request).

Whenever you try to use one of the subkeys (or the master key), rather than being asked for the passphrase for the key (which you should have set as a very difficult, absolutely unguessable string that you and only you could remember and think of), you will be asked to enter the User PIN set for the smartcard.

You've achieved proper two-factor authentication.

Note of ECC on the Yubikey: according to the marketing documentation, the Yubikey knows about ECC p256 and ECC p384. Unfortunately, it looks like safecurves.cr.yp.to considers these unsafe, since they do not meet all the SafeCurves requirements. I'm not especially versed in cryptography, but this means I'll read up more on the subject, and stay away from the ECC implementation on the Yubikey 4 for now. However, it doesn't seem, at first glace, that this ECC implementation is meant for GPG at all. The Yubikey also has PIV magic which would allow it to be used as a pure SSH smartcard (rather than using a GPG authentication subkey for SSH), with a private certificate being generated by the card. These certificates could be created using RSA or ECC. I tried to play a bit with it (using RSA), following the SSH with PIV and PKCS11 document on developers.yubico.com; but I didn't manage to make it work. It looks like the GPG functions might interfere with PIV in some way or I could just not handle the ssh-agent the right way. I'm happy to be shown how to use this correctly.

Checklist announcement

I've been running EC2 AMI tests for a little while now, and it quickly became apparent
that many if not all of the tests could be automated and would then probably be much more
interesting to run -- that, and just the automation process is really interesting to work on.

At the moment, it's only a branch that I've registered in its own
project: lp:checklist. Its project page is here: https://edge.launchpad.net/checklist

You can get started using checklist by grabbing it from Bazaar (since there is no official
release yet):

bzr branch lp:checklist

It's possible that a lot of the work I've done could just as well have been included into other
tools, like checkbox, but for starters, I was mostly interested in getting some basic stuff to
work, and worrying about the other details later.


At this point, checklist is able to run commands over ssh, locally, and the most interesting
feature is that it can also create EC2 instances (using python-boto), which it would then ssh
onto to run a testcase.

Checklist also is all configured using an easy to use configuration file format: ini files. It
can look at the stdout and stderr of the commands run in order to check for success or failure
using regexes, which will give a fairly high level of control to someone writing special tests.

Thinking about it more, I'm also going to be using it to remotely verify machines that have been
kickstarted to make sure the unattended installs run properly and do everything required. The
fact that it can run tests on a remote machine is definitely a plus when trying to test
kickstarted systems from an isolated network.

Don't hesitate to branch checklist and provide me with patches :)

My Ubuntu development tools

This is another post about my work towards becoming MOTU.

I've been working a lot on various packages, such as network-manager, concordance, and more recently acct and acct-ng. On the way, you always get to know more, and following the spirit of open source, I've decided to share what I've learned.

Since I've started to do packaging work, I've been keeping various little configuration files around to tweak the behaviour of the various utilities we get to use daily. I've been keeping all those in a 'junk' bazaar repository, but it's getting much less like 'junk' and more like something actually usable, so I'm publishing it:

lp:~mathieu-tl/+junk/development-tools

It's a branch new branches that contains the very latest version of all the config I've been using, including some aliases I usually always have set on my systems. I will be keeping all the files up to date there from now on, so you will likely see things evolve over time.

Here's what I have in there:

• A .pbuilderrc file: it allows me to specify DIST=karmic (or sid or whatever else) at the command line, and also makes use of mini-dinstall to quickly build packages and changes the packages available when running builds. As such, the following command will build a package for karmic:

sudo DIST=karmic pbuild build glittering_new_package.dsc

Just as this line will build the same package for jaunty (provided all is well):

sudo DIST=jaunty pbuild build glittering_new_package.dsc

• .mini-dinstall.conf : The second part to the above. Specifies some rules for mini-dinstall (requires creating /var/cache/archive...)
• .dput.cf has the last part of this stuff, or where to put packages, and how to let mini-dinstall know about them
• .dput.cf also knows how to push source packages to PPAs, as well as to mentors.debian.org.
• .quiltrc : to make pretty patches. Then you can make them follow DEP3 too.
• .selected_editor : because I just really love vim.
• .bazaar/builddeb.conf : tweaks to directories used by bzr bd.
• .ssh/config : very nice and time-saving tweaks to SSH.
• .shell* files : they contain additional aliases, shell functions, and variables that are either extremely useful (at least to me) or just plain mandatory for anyone serious in doing packaging, such as DEBFULLNAME, DEBEMAIL, or BZR_EMAIL.

Oh yeah, and you might want to change those last three variables to your own values in .shell_custom_vars... :)

All that includes a quick README file, but essentially it's a matter of dropping the files in your home directory and logging in and out, or closing and reopening a terminal. All the shell stuff works by including .shell_custom_start somewhere in your profile.

If you find stuff that you think is totally wrong or needs more work, please let me know. There is still a lot of room for improvement.

Process Accounting data

I've been working since yesterday on a very interesting project: dealing with process accounting data, in the context of PCI-DSS compliance.

Process accounting is quite interesting because of how it's done. Here's a gross over-simplification, so sorry if I'm explaining it all wrong: the kernel, if it has "BSD Process accounting" enabled (and/or "version 3", as it is in Ubuntu, or at least on my Karmic system), waits for a special signal for userland to turn on process accounting and start writing binary data to a file on the filesystem.

But let's back up a little. What is this process accounting stuff?

Process accounting is a feature that allows you to track every command ever run on a system by any user. As the name says, it helps to "account" for all actions on a system and potentially know about actions that should not have occurred (say, a system that was compromised), or a command that caused an outage and it must be tracked down to who was responsible for an evil evil intervention on a production system without advising anybody... it requires both a kernel piece and a userland piece.

By default, on most systems, the kernel will already have everything enabled to support process accounting. You will only need to install the userland software.

On Ubuntu, you can achieve this with the command:

sudo apt-get install acct

On Fedora system, you can get the same result with this command:

yum install psacct

Once that is done, you will run 'sudo /etc/init.d/acct start' on Ubuntu, or 'service psacct start' on Fedora to start the process accounting (give the go to the kernel to write in the data file).

This is where things get complicated. I've searched a little for newer, more versatile tools to handle the binary data spewed out by my systems. It turns out that while acct (or psacct), in other words the GNU Accounting Utilities are great, they tend to be the only option for dealing with the kernel's data, and pretty much also force you to physically connect to a system in order to search for the information you want. Needless to say, this can tend to be problematic (say the system is now completely broken... what can you do?). It also brings up the issue that in the event of a compromise of the system, you can no longer trust the system, or the files it contains -- not even the accounting data. Backups are an option, so is rsync, but we were looking for something better. Something that would send to syslog for example, to be integrated into Splunk or some alerting utilities, to both centralize and potentially render the data as read-only as it could get.

That's where my scripts come in.

I've been busy writing a replacement for the lastcomm command as well as something to grab the data from the binary file and feed it into syslog. I've called it acct2syslog.pl.

This is still very early and experimental work, but I already have a Bazaar branch for it. Check it the Launchpad page! If you're interested in helping out, feel free to send me an email or message on IRC.

Obviously, I can't account for performance at this time. I just don't know how well it will deal with a system if there are thousands of commands running really fast. :)

Visite du projet Byobu

Si vous utilisez souvent votre système Linux en mode texte, vous connaissez probablement GNU/Screen. Il s'agit d'un gestionnaire de fenêtre texte très avancé qui permet de démarrer une session, la détacher, puis la rattacher plus tard et revenir où la session était rendue -- que ce soit directement sur la même machine, ou alors à distance. Pour plus d'information sur GNU/Screen lui-même, consultez la Présentation de GNU/Screen, par Labo Linux.

Si vous utilisez régulièrement Screen, vous serez heureux d'apprendre que celui-ci a été agrémenté officiellement depuis Jaunty d'un paquet voué à son extension: screen-profiles.

screen-profiles, créé par Dustin Kirkland, développeur pour Ubuntu, est un amalgame d'un nombre de scripts voués à rendre plus simple l'utilisation de screen, ainsi qu'à fournir à ses utilisateurs de nombreux petits outils pour simplifier la vie, et rendre la configuration de screen plus simple.

screen-profiles (et son supplément, screen-profiles-extras) peuvent être installés par la commande suivante:

sudo apt-get install screen-profiles screen-profiles-extras

Une fois installé, vous pourrez lancer screen; et après avoir choisi le profile de départ, appuyez sur F9 pour pouvoir configurer l'interface. Les deux lignes du bas servent à présenter les fenêtres possibles et les notifications.


Le menu de configuration permet d'afficher l'aide, modifier le thème utilisé, les raccourcis clavier, les notifications au bas de l'écran, les fenêtres et les paramètres de démarrage.



Par défaut, trois thèmes sont disponibles avec le paquet screen-profiles: light, dark, et black. Les thèmes additionnels présentés dans l'écran ci-dessous proviennent de screen-profiles-extras.



Après modification des options, vous devrez appuyer sur F5 puis Enter pour recharger la configuration.





Le résultat est affiché ici, avec le thème dark-blue.

Enfin, d'ici la sortie de Karmic, les paquets à utiliser seront renommés à byobu et byobu-extras suivant le changement de nom du projet screen-profiles pour byobu...

Vous trouverez la page Launchpad du projet ici: https://launchpad.net/byobu

How-to fix SVN's bug 160004: corrupted file system.

My notes on the subject... The idea is generally to revert to the last sucessful commit, then work from there. It's less than ideal, but at least it's relatively safe to do, and easy.

First, find the commit that broke things:

svnlook youngest /path/to/repository

Next, check whether the commit was successful, and proceed on backwards in revision numbers until you find one that works:

svnadmin dump /path/to/repository -r rev_number_from_svnlook --incremental > /home/mtrudel/rev_dump.svndump

If it works, you're stuck. If it fails, and it should, continue on to the previous build (current_build - 1), if it succeeds, you can do the following:

• Edit /path/to/repository/db/current, change the first item (revision number) to the revision that works.
• Try to checkout -- things should work.
• Try to checkin changes -- things should work.

You should obviously do this with exclusive access to the SVN repository you're trying to fix, *just in case*, so shutdown Apache, svnserve or whatever else may be publishing the repository.

Other details:

• svnadmin verify is a bad idea on repos with a large number of commits, if resolution time is critical and the information in the last commit is not too important.
• Take backups of any files you modify.

One other little known command

Do you know about "join"? It's a nice little utility that comes from the "coreutils" package, which means that it's there by default on most, if not all Unix/Linux distributions. What this does is join (duh) two files, based on the values in fields.

For example, the other day I had to correlate Apache requests from the logs on one Apache server acting as a proxy, with the logs of another Apache server used as a backend. Because of missing configuration, the proxy Apache had public IP addresses, but didn't have domain names (virtual hosts), and the backend had domain names but only the private IP addresses from the proxy systems. So, I took copies of the two Apache's combined_logs, and awk'ed them to only keep the IP, URI requested, and domain name requested (this was necessary because it's better to have clear field numbers with join, and Apache can, if configured to do it, log the UserAgent of a client, which spans multiple fields if you field separator is space).

With the logs doctored as required to be easy to handle with join, you can just run the command:

join -1 2 -2 2 -o "1.1 0 2.3" log_proxy.txt log_backend.txt

What this will do, is tell join to use the second field from the first file (-1 2, aka log_proxy.txt), the second field from the second file (-2 2, aka log_backend), and join the data together to form a new file, outputted to standard output, following the format stated in -o: first file's first field (the public IP), the join field (0, or the URI in this case, which appears in both files), and the third field in the second file (the virtual host domain). You can obviously adjust the field numbers if necessary, or change various settings such as the field separator (default is space), which is all very clearly documented in the man page.

On other news, I seem to have because fairly well know in my workplace to be the Ubuntu wizard... Maybe the Ubuntu laptop bag helps? Anyway... Friday I was asked by Marc about installing Ubuntu on a Lenovo T60, and yesterday I was asked about my thoughts and experiences with Intrepid by Richard. I've already managed to get one Ubuntu machine in use as a server, and hopefully there will be more to come in the future.

I learned a new command today

Let's make it simple, it's pretty self-explanatory. Since I'm a netadmin, every little thing can help, and this is no exception. It should make my life MUCH easier.

sudo apt-get install sipcalc

And then:

mtrudel@icarus:~$ sipcalc 10.10.0.0/21
-[ipv4 : 10.10.0.0/21] - 0

[CIDR]
Host address - 10.10.0.0
Host address (decimal) - 168427520
Host address (hex) - A0A0000
Network address - 10.10.0.0
Network mask - 255.255.248.0
Network mask (bits) - 21
Network mask (hex) - FFFFF800
Broadcast address - 10.10.7.255
Cisco wildcard - 0.0.7.255
Addresses in network - 2048
Network range - 10.10.0.0 - 10.10.7.255
Usable range - 10.10.0.1 - 10.10.7.254

-
mtrudel@icarus:~$ sipcalc 10.10.9.9
-[ipv4 : 10.10.9.9] - 0

[CIDR]
Host address - 10.10.9.9
Host address (decimal) - 168429833
Host address (hex) - A0A0909
Network address - 10.10.9.9
Network mask - 255.255.255.255
Network mask (bits) - 32
Network mask (hex) - FFFFFFFF
Broadcast address - 10.10.9.9
Cisco wildcard - 0.0.0.0
Addresses in network - 1
Network range - 10.10.9.9 - 10.10.9.9

-

Beautiful little program, gives you all kind of useful information, and it's free!