Tuesday, 20 June 2017

Netplan by default in 17.10

Friday, I uploaded an updated nplan package (version 0.24) to change its Priority: field to important, as well as an update of ubuntu-meta (following a seeds update), to replace ifupdown with nplan in the minimal seed.

What this means concretely is that nplan should now be installed by default on all images, part of ubuntu-minimal, and dropped ifupdown at the same time.

For the time being, ifupdown is still installed by default due the way debootstrap generates the very minimal images used as a base for other images -- how it generates its base set of packages, since that depends only on the Priority: field of packages. Thus, nplan was added, but ifupdown still needs to be changed (which I will do shortly) to disappear from all images.

The intent is that nplan would now be the standard way of configuring networks. I've also sent an email about this to ubuntu-devel-announce@.

I've already written a bit about what netplan is and does, and I have still more to write on the subject (discussing syntax and how to do common things). We especially like how using a purely declarative syntax makes things easier for everyone (and if you can't do what you want that way, then it's a bug you should report).

MaaS, cloud-init and others have already started to support writing netplan configuration.

The full specification (summary wiki page and a blueprint reachable from it) for the migration process is available here.

While I get to writing something comprehensive about how to use the netplan YAML to configure networks, if you want to know more there's always the manpage, which is the easiest to use documentation. It should always be up to date with the current version of netplan available on your release (since we backported the last version to Xenial, Yakkety, and Zesty), and accessible via:

man 5 netplan

To make things "easy" however, you can also check out the netplan documentation directly from the source tree here:

https://git.launchpad.net/netplan/tree/doc/netplan.md

There's also a wiki page I started to get ready that links to the most useful things, such as an overview of the design of netplan, some discussion on the renderers we support and some of the commands that can be used.

We even have an IRC channel on Freenode: #netplan

I think you'll find that using netplan makes configuring networks easy and even enjoyable; but if you run into an issue, be sure to file a bug on Launchpad here:

Wednesday, 24 May 2017

An overview of UEFI Secure Boot on Ubuntu

Secure Boot is here

Ubuntu has now supported UEFI booting and Secure Boot for long enough that it is available, and reasonably up to date, on all supported releases. Here is how Secure Boot works.

An overview

I'm including a diagram here; I know it's a little complicated, so I will also explain how things happen (it can be clicked to get to the full size image).


In all cases, booting a system in UEFI mode loads UEFI firmware, which typically contains pre-loaded keys (at least, on x86). These keys are usually those from Microsoft so that Windows can load its own bootloader and verify it, as well as those from the computer manufacturer. The firmware doesn't, by itself, know anything special about how to boot the system -- this is something that is informed by NVRAM (or some similar memory that survives a reboot) by way of a few variables: BootOrder, which specified what order to boot things in, as well as BootEntry#### (hex numbers), which contains the path to the EFI image to load, a disk, or some other method of starting the computer (such as booting in the Setup tool for that firmware). If no BootEntry variable listed in BootOrder gets the system booting, then nothing would happen. Systems however will usually at least include a path to a disk as a permanent or default BootEntry. Shim relies on that, or on a distro, to load in the first place.

Once we actually find shim to boot; this will try to validate signatures of the next piece in the puzzle: grub2, MokManager, or fallback, depending on the state of shim's own variables in NVRAM; more on this later.

In the usual scenario, shim will validate the grub2 image successfully, then grub2 itself will try to load the kernel or chainload another EFI binary, after attempting to validate the signatures on these images by way of asking shim to check the signature.

Shim

Shim is just a very simple layer that holds on to keys outside of those installed by default on the system (since they normally can't be changed outside of Setup Mode, and require a few steps to do), and knows how to load grub2 in the normal case, as well as how to load MokManager if policy changes need to be applied (such as disabling signature validation or adding new keys), as well as knowing how to load the fallback binary which can re-create BootEntry variables in case the firmware isn't able to handle them. I will expand on MokManager and fallback in a future blog post.

Your diagram says shim is signed by Microsoft, what's up with that?

Indeed, shim is an EFI binary that is signed by Microsoft how we ship it in Ubuntu. Other distributions do the same. This is required because the firmware on most systems already contains Microsoft certificates (pre-loaded in the factory), and it would be impractical to have different shims for each manufacturer of hardware. All EFI binaries can be easily re-signed anyway, we just do things like this to make it as easy as possible for the largest number of people.

One thing this means is that uploads of shim require a lot of effort and testing. Fortunately, since it is used by other distributions too, it is a well-tested piece of code. There is even now a community process to handle review of submissions for signature by Microsoft, in an effort to catch anything outlandish as quickly and as early as possible.

Why reboot once a policy change is made or boot entries are rebuilt?

All of this happens through changes in firmware variables. Rebooting makes sure we can properly take into account changes in the firmware variables, and possibly carry on with other "backlogged" actions that need to happen (for instance, rebuilding BootEntry variables first, and then loading MokManager to add a new signing key before we can load a new grub2 image you signed yourself).

Grub2

grub2 is not a new piece of the boot process in any way. It's been around for a long while. The difference from booting in BIOS mode compared to in UEFI is that we install an UEFI binary version of grub2. The software is the same, just packaged slightly differently (I may outline the UEFI binary format at some point in the future). It also goes through some code paths that are specific to UEFI, such as checking if we've booting through shim, and if so, asking it to validate signatures. If not, we can still validate signatures, but we would have to do so using the UEFI protocol itself, which is limited to allowing signatures by keys that are included in the firmware, as expressed earlier. Mostly just the Microsoft signatures.

grub2 in UEFI otherwise works just like it would elsewhere: it try to find its grub.cfg configuration file, and follow its instructions to boot the kernel and load the initramfs.

When Secure Boot is enabled, loading the kernel normally requires that the kernel itself is signed. The kernels we install in Ubuntu are signed by Canonical, just like grub2 is, and shim knows about the signing key and can validate these signatures.

At the time of this writing, if the kernel isn't signed or is signed by a key that isn't known, grub2 will fall back to loading the kernel as a normal binary (as in not signed), outside of BootServices (a special mode we're in while booting the system, normally it's exited by the kernel early on as the kernel loads). Exiting BootServices means some special features of the firmware are not available to anything that runs afterwards, so that while things may have been loaded in UEFI mode, they will not have access to everything in firmware. If the kernel is signed correctly, then grub2 leaves the ExitBootServices call to be done by the kernel.

Very soon, we will stop allowing to load unsigned (or signed by unknown keys) kernels in Ubuntu. This is work in progress. This change will not affect most users, only those who build their own kernels. In this case, they will still be able to load kernels by making sure they are signed by some key (such as their own, and I will cover signing things in my next blog entry), and importing that key in shim (which is a step you only need to do once).

The kernel

In UEFI, the kernel enforces that modules loaded are properly signed. This means that for those who need to build their own custom modules, or use DKMS modules (virtualbox, r8168, bbswitch, etc.), you need to take more steps to let the modules load properly.

In order to make this as easy as possible for people, for now we've opted to let users disable Secure Boot validation in shim via a semi-automatic process. Shim is still being verified by the system firmware, but any piece following it that asks shim to validate something will get an affirmative response (ie. things are valid, even if not signed or signed by an unknown key). grub2 will happily load your kernel, and your kernel will be happy to load custom modules. This is obviously not a perfectly secure solution, more of a temporary measure to allow things to carry on as they did before. In the future, we'll replace this with a wizard-type tool to let users sign their own modules easily. For now, signature of binaries and modules is a manual process (as above, I will expand on it in a future blog entry).

Shim validation

To toggle shim validation, if you were using DKMS packages and feel you'd really prefer to have shim validate everything (but be aware that if your system requires these drivers, they will not load and your system may be unusable, or at least whatever needs that driver will not work):
sudo update-secureboot-policy --enable
If nothing happens, it's because you already have shim validation enabled: nothing has required that it be disabled. If things aren't as they should be (for instance, Secure Boot is not enabled on the system), the command will tell you.

And although we certainly don't recommend it, you can disable shim validation yourself with much the same command (see --help). There is an example of use of update-secureboot-policy here.

Tuesday, 23 May 2017

ss: another way to get socket statistics

In my last blog post I mentioned ss, another tool that comes with the iproute2 package and allows you to query statistics about sockets. The same thing that can be done with netstat, with the added benefit that it is typically a little bit faster, and shorter to type.

Just ss by default will display much the same thing as netstat, and can be similarly passed options to limit the output to just what you want. For instance:

$ ss -t
State       Recv-Q Send-Q       Local Address:Port                        Peer Address:Port              
ESTAB       0      0                127.0.0.1:postgresql                     127.0.0.1:48154              
ESTAB       0      0            192.168.0.136:35296                      192.168.0.120:8009                
ESTAB       0      0            192.168.0.136:47574                     173.194.74.189:https
[...]

ss -t shows just TCP connections. ss -u can be used to show UDP connections, -l will show only listening ports, and things can be further filtered to just the information you want.

I have not tested all the possible options, but you can even forcibly close sockets with -K.

One place where ss really shines though is in its filtering capabilities. Let's list all connections with a source port of 22 (ssh):

$ ss state all sport = :ssh
Netid State      Recv-Q Send-Q     Local Address:Port                      Peer Address:Port              
tcp   LISTEN     0      128                    *:ssh                                  *:*                  
tcp   ESTAB      0      0          192.168.0.136:ssh                      192.168.0.102:46540              
tcp   LISTEN     0      128                   :::ssh                                 :::* 
And if I want to show only connected sockets (everything but listening or closed):

$ ss state connected sport = :ssh
Netid State      Recv-Q Send-Q     Local Address:Port                      Peer Address:Port              
tcp   ESTAB      0      0          192.168.0.136:ssh                      192.168.0.102:46540 

Similarly, you can have it list all connections to a specific host or range; in this case, using the 74.125.0.0/16 subnet, which apparently belongs to Google:

$ ss state all dst 74.125.0.0/16
Netid State      Recv-Q Send-Q     Local Address:Port                      Peer Address:Port              
tcp   ESTAB      0      0          192.168.0.136:33616                   74.125.142.189:https              
tcp   ESTAB      0      0          192.168.0.136:42034                    74.125.70.189:https              
tcp   ESTAB      0      0          192.168.0.136:57408                   74.125.202.189:https
        
This is very much the same syntax as for iptables, so if you're familiar with that already, it will be quite easy to pick up. You can also install the iproute2-doc package, and look in /usr/share/doc/iproute2-doc/ss.html for the full documentation.

Try it for yourself! You'll see how well it works. If anything, I'm glad for the fewer characters this makes me type.

Tuesday, 9 May 2017

If you're still using ifconfig, you're living in the past

The world evolves

I regularly see "recommendations" to use ifconfig to get interface information in mailing list posts or bug reports and other places. I might even be guilty of it myself. Still, the world of networking has evolved quite a lot since ifconfig was the de-facto standard to bring up a device, check its IP or set an IP.

Following some improvements in the kernel and the gradual move to driving network things via netlink; ifconfig has been largely replaced by the ip command.

Running just ip yields the following:

Usage: ip [ OPTIONS ] OBJECT { COMMAND | help }
       ip [ -force ] -batch filename
where  OBJECT := { link | address | addrlabel | route | rule | neigh | ntable |
                   tunnel | tuntap | maddress | mroute | mrule | monitor | xfrm |
                   netns | l2tp | fou | macsec | tcp_metrics | token | netconf | ila }
       OPTIONS := { -V[ersion] | -s[tatistics] | -d[etails] | -r[esolve] |
                    -h[uman-readable] | -iec |
                    -f[amily] { inet | inet6 | ipx | dnet | mpls | bridge | link } |
                    -4 | -6 | -I | -D | -B | -0 |
                    -l[oops] { maximum-addr-flush-attempts } | -br[ief] |
                    -o[neline] | -t[imestamp] | -ts[hort] | -b[atch] [filename] |
                    -rc[vbuf] [size] | -n[etns] name | -a[ll] | -c[olor]}

I understand this may look complicated to some people, but the jist of it is to understand that with ip, you interact with objects, and apply some kind of function to it. For example:

ip address show

This is the main command that would be used in place of ifconfig. It will just display the IP addresses assigned to all interfaces. To be precise, it will show you the layer 3 details the interface: the IPv4 and IPv6 addresses, whether it is up, what are the different properties related to the addresses...

Another command will give you details about the layer 2 properties of the interface: its MAC address (ethernet address), etc; even if it is shown by ip address:

ip link show

Furthermore, you can set devices up or down (similar to ifconfig eth0 up or ifconfig eth0 down) simply by using:

ip link set DEVICE up or ip link set DEVICE down

As shown above, there are lots of other objects that can be interacted with using the ip command. I'll cover another: ip route, in another post.

Why is this important?

As time passes, more and more features are becoming easier to use with the ip command instead of with ifconfig. We've already stopped installing ifconfig on desktops (it still gets installed on servers for now), and people have been discussing dropping net-tools (the package that ships ifconfig and a few other old commands that are replaced) for a while now. It may be time to revisit not installing net-tools by default anywhere.

I want to know about your world

Are you still using one of the following tools?

/bin/netstat    (replaced by ss, for which I'll dedicate another blog post entirely)
/sbin/ifconfig
/sbin/ipmaddr   (replaced by ip maddress)
/sbin/iptunnel
/sbin/mii-tool    (ethtool should appropriately replace it)
/sbin/nameif
/sbin/plipconfig
/sbin/rarp
/sbin/route
/sbin/slattach

If so and there is just no alternative to using them that comes from iproute2 (well, the ip or ss commands) that you can use to do the same, I want to know about how you are using them. We're always watching for things that might be broken by changes; we want to avoid breaking things when possible.

Friday, 5 May 2017

Quick and easy network configuration with Netplan

Earlier this week I uploaded netplan 0.21 in artful, with SRUs in progress for the stable releases. There are still lots of features coming up, but it's also already quite useful. You can already use it to describe typical network configurations on desktop and servers, all the way to interesting, complicated setups like bond over a bridge over multiple VLANs...

Getting started

The simplest netplan configuration might look like this:

# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager
At boot, netplan will see this configuration (which happens to be installed already on all new systems since 16.10) and generate a single , empty file: /run/NetworkManager/conf.d/10-globally-managed-devices.conf. This tells the system that NetworkManager is the only renderer for network configuration on the system, and will manage all devices by default.

Working from there: a simple server

Let's look at it on a hypothetical web server; such as for my favourite test: www.perdu.com.

network:
  version: 2
  ethernets:
    eth0:
      dhcp4: true
This incredibly simple configuration tells the system that the eth0 device is to be brought up using DHCP4. Netplan also supports DHCPv6, as well as static IPs, setting routes, etc.


Building up to something more complex

Let's say I want a team of two NICs, and use them to reach VLAN 108 on my network:

            network:
              version: 2
              ethernets:
                eth0:
                  dhcp4: n
                eth1:
                  mtu: 1280
                  dhcp4: n
              bonds:
                bond0:
                  interfaces:
                  - eth1
                  - eth0
                  mtu: 9000
              vlans:
                bond0.108:
                  link: bond0
                  id: 108

I think you can see just how simple it is to configure even pretty complex networks, all in one file. The beauty in it is that you don't need to worry about what will actually set this up for you.

A choice of backends

Currently, netplan supports either NetworkManager or systemd-networkd as a backend. The default is to use systemd-networkd, but given that it does not support wireless networks, we still rely on NetworkManager to do just that.

This is why you don't need to care what supports your config in the end: netplan abstracts that for you. It generates the required config based on the "renderer" property, so that you don't need to know how to define the special device properties in each backend.

As I mentioned previously, we are still hard at work adding more features, but the core is there: netplan can set up bonds, bridges, vlans, standalone network interfaces, and do so for both static or DHCP addresses. It also supports many of the most common bridge and bond parameters used to tweak the precise behaviour of bonded or bridged devices.


Coming up...

I will be adding proper support for setting a "cloned" MAC on a device. I'm reviewing the code already to do this, and ironing out the last issues.

There are also plans on better handling administrative states for devices; along with a few bugs that relate to support MaaS, where having a simple configuration style really shines.

I'm really excited for where netplan is going. It seems like it has a lot of potential to address some of the current shortcomings in other tools. I'm also really happy to hear of stories of how it is being used in the wild, so if you use it, don't hesitate to let me know about it!

Contributing

All of the work on netplan happens on Launchpad. Its source code is at https://code.launchpad.net/netplan; we always welcome new contributions.

Thursday, 2 June 2016

Netflix, or the pains of dealing with royalties and DMCA

A few days ago, after enjoying the use of a pretty much static IP address for a long while from my ISP (it hadn't changed in easily a year), my IP changed. This took down my IPv6 tunnel, which I tend to use a lot to access to various services for work -- you know, dogfooding and all of that. My IPv6 address depends on a tunnel that needs to stay up (and for that requires my IPv4 address to not change too much, but whatever).

Probably since then (but I did not really notice until yesterday or so), I've had multiple issues with Netflix streaming. As many know, Netflix is now enforcing some method of trying to detect VPN and proxy users to force local content upon its users. I think it's a stupid idea, but I see where they come from with that decision.

Netflix has to deal with royalties, copyright, and varying laws depending on where a user might be. For instance, you may wish to watch NCIS -- this will likely depend on Netflix having paid CBS (or whomever the title belong to) to be allowed to present it to clients. I have no idea how these costs are done, they might well be a percentage based on number of viewers or some such.

In the US, this is relatively easy, they can deal with local companies and handle things. This becomes more complicated when you factor in different copyright laws in other countries, and exclusivity rights, etc. In the case of NCIS, Global appears to have (exclusive?) rights for NCIS, so they look to be the only legit place to stream episodes online. I suspect Netflix would possibly have to pay *them* to stream NCIS in Canada, or otherwise be subject to random other byzantine rules. I don't pretend to understand the intricacies past the one class I took on Canadian copyright/patent/IP law over a year ago. Suffice to say it's complicated, and there are probably good reasons to try and have users in country X watch country X's content, and not country Y's. It's likely to cut costs.

My issue stemmed from the fact that with the reset of my IPv4/IPv6 connection, or possibly just as a coincidence, Netflix started to care about my IPv6 addresses. It's possible that geoip data informed this, or that Netflix started to do more checking, or started to do proper IPv6, etc. I don't know.

I had an online chat with an awesome Netflix Customer Service rep; HecThor (the name is awesome too!), and received great service even if they could not help:


Netflix Customer Service
You are now chatting with: Hecthor
Netflix Hecthor
Hello!! My name is HecThor! How can I help?
You
Hi, I'm Matt, I keep getting error messages saying that I am behind a proxy or VPN when I am not
You
Would you be able to consult logs or whatever you might have to tell me why that has been detected so I can take the necessary steps?
Netflix Hecthor
Oh, let me check this out for you. Could you please tell me which device are you using?
You
Right now, my Chrome browser
You
probably listed as Chrome on linux, version 50.0.2661.94
You
I had the same issues on a different device too (another Chrome, version is most likely different as it is running on a Chromebook device)
Netflix Hecthor
Just a quick question, have you tried going to the extensions of Google Chrome and unchecked and tried Netflix one more time?
You
yes
You
what's more, this one does not have any extensions
Netflix Hecthor
Alright, just to confirm, are you using Linux?
You
not exclusively, but yes
Netflix Hecthor
Oh got it, I'm seeing here that the signal is being redirected to the US and then to Canada several times in a day, in this case the best thing to do is to check with your Internet Service Provider to investigate why your connection appears to come from a VPN or a proxy service, as they are in charge of the signal.
You
this is to be expected, I get IPv6 connectivity from a US provider for work purposes
You
could it be that you guys started to allow ipv6?
Netflix Hecthor
Oh got it, we do support with IPv6, however having the setting set to the US instead of Canada may cause this conflict , so in this case what I recommend is to turn it off and you'll be able to stream without a problem. :)
You
it's not the kind of thing I can turn off
You
there aren't providers here who do IPv6
You
is there any way for you to set my account to only use IPv4?
Netflix Hecthor
Got it, you see we don't have a way to set an account to use IPv4 or Ipv6 as this has to do with the Internet service, so in this case I would recommend you to contact them and try to reset the signal or check if they're able to do that change on your settings, I'm sure that once they do you'll be able to stream Netflix without a problem.
You
There is no thing to reset, there is no Canada endpoint for this thing.
You
in fact, it only started to be an issue since the last reset, because my IPv4 address changed a few days ago as well
Netflix Hecthor
I understand, and do you have a way so you can return to IPv4? The thing is that Netflix is working fine, however the system is detecting that your IP is constantly changing from region to region, this is why the system is not letting you stream.
You
I can't do this change on the local systems, no. This is how my home network is set up -- like I said, I do need IPv6 for my work. I work from home.
Netflix Hecthor
Oh I definitely understand what you mean, however, to be completely honest, the process you use will not let you stream. Unless you change that wont be able to stream, because when the system detects that you're in a country and your network shows another one, this issue appears, it might work some times but I can't guaranty it will always work, if you like you can try Netflix on your mobile's network to verify this.
You
I don't especially want to verify anything, since we have a fair expectation of what the issue is
You
you've been quite helpful
You
Do you object to me using this chat log for documentation purposes?
You
I can remove your name if you prefer, but I thought it looked badass enough ;)
Netflix Hecthor
Sure, no problem, and it's been a pleasure being able to help! :) Is there anything else I can do for you?

I went on to ask to file a complaint / provide feedback to the team, since Netflix should be aware of the complexity, and inconvenience this poses on its customers. Still, I want to reiterate that I was quite happy with the service I've had from Customer Support rep HecThor, who was helpful and understanding.

I'm technical enough to be able to deal with such issues in various ways. I did some searching, and it looks like you *can not* disable IPv6 simply for Chrome. It's also impractical to disable the IPv6 tunnel... I have it up for a good reason, and it had been working for a long while (that too, over a year) with no issues. Other people could also have other special network setups that could impede on Netflix steaming services. VPNs happen, and they are not all used to watch US content. They can also be done at the router level rather than at the device level; and even some ISPs require PPTP VPN use to get any kind of connectivity at all (or did in the past).

The inability to disable IPv6 in Chrome is probably really a usability bug in it, but it shows how the average user might eventually run into issues dealing with content "blocking" based on location. I'm not really expecting the average user to have a network setup like mine: I had to set up IPv6 myself here, as none of the providers in Canada do a satisfactory job at it. I also don't expect the average user to care about the IP family at all -- but we'll soon get to a point where blocking based on IP and location won't make sense. IPv6 is meant to improve mobility, and there are some steps taken to ensure this (see RFC 3775). GeoIP data can be wrong, misleading, or simply inexistant too, so you really ought not to rely on that at all.

Netflix has been doing relatively well in leading some interesting infrastructure ideas it seems, aside from not being very cooperative with Linux users for a long while (fortunately, now Netflix works on Linux, but only with the official Google Chrome, still not with free software browsers). It would be good to see that leadership continue and avoid restrictive policies in favor of cooperation, especially for a company priding itself on using Linux and open source technologies.

For now, I've opted to null-route IPv6 netflix, which means I get a small delay but I can still watch Futurama. It's the least intrusive change I thought of to not have to tear down my IPv6 tunnel, but still be able to watch content.

If for some inexplicable reason you also have a Cisco router at home and use an IPv6 provider from the US to get IPv6 connectivity and want to make sure Netflix keeps working; this is the command I used:

#ipv6 route 2406:DA00:FF00::/48 Null0

Rather than using outdated, unreliable technology to enforce restrictive, ill-designed content rules, Netflix should lead an overhaul of the limitations imposed upon it by the original content providers. That, or use some of those uncountable piles of moneys to cover potential costs of out-of-country-content access.

Monday, 29 February 2016

Nominations wanted for the Developer Membership Board

Hi!

The Ubuntu Developer Membership Board is in need of new blood.

Of the seven members of the board, five (5) will be expiring on March 9th. Members of the Developer Membership Board are elected by all Ubuntu Developers for a term of 2 years, meeting in #ubuntu-meeting about once a fortnight. Candidates should be Ubuntu developers themselves, and should be well qualified to evaluate prospective Ubuntu developers.

The DMB is responsible for reviewing developer applicants and decides when to entrust them with developer privileges or to grant them Ubuntu membership status.

Providing at least six valid nominations are received, the new members will be chosen using Condorcet voting. Members of the ubuntu-dev team in Launchpad will be eligible to vote, and will receive voting ballots by email (to their email address recorded in Launchpad). A Call for Nominations has already been sent by email to the ubuntu-devel-announce mailing list (but another call for nominations should follow soon): https://lists.ubuntu.com/archives/ubuntu-devel-announce/2016-February/001167.html.

Applications should be sent as GPG-signed emails to developer-membership-board at lists.ubuntu.com (which is a private mailing list accessible only by DMB members).

Of course, if you're nominating a developer other than yourself, please make sure to ask who you're about to nominate beforehand, to make sure they're okay with it.