JADDOG

On TryStack.org we have an automated script that cleans off instances after they’ve run for 24 hours.

We also allow people to attach volumes to their instances. In our script to delete the long running instances we naively thought we could just call delete() on an instance and it would all be cleaned up, not so.

The instances get stuck in a “deleting” state and the instances and the volumes aren’t able to be cleaned up. The compute node has actually released the iscsi target cinder presented to it:

[root@host11 ~]# iscsiadm -m session
iscsiadm: No active sessions.

though, tgtd hasn’t released the lvm device for some reason, so the device can’t be deleted:

[root@host2 ~]# lvremove cinder-volumes/volume-b9869d42-418f-4d7c-b4bf-951b035d1817
Do you really want to remove active logical volume volume-b9869d42-418f-4d7c-b4bf-951b035d1817? [y/n]: y
device-mapper: remove ioctl on failed: Device or resource busy
Unable to deactivate cinder–volumes-volume–b9869d42–418f–4d7c–b4bf–951b035d1817 (253:59)
Unable to deactivate logical volume “volume-b9869d42-418f-4d7c-b4bf-951b035d1817″
[root@host2 ~]# lsof /dev/cinder-volumes/volume-b9869d42-418f-4d7c-b4bf-951b035d1817
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
tgtd 10812 root 74u BLK 253,59 0t0 45476393 /dev/cinder-volumes/../dm-59

To fix this use tgt-admin to delete the target, there by relinquishing tgtd’s use of the volume and then clean up the cinder entry in the database so that OpenStack think’s it can now do the needful. (and actually can now!)

[root@host2 ~]# tgt-admin -s | grep b98
Target 61: iqn.2010-10.org.openstack:volume-b9869d42-418f-4d7c-b4bf-951b035d1817
Backing store path: /dev/cinder-volumes/volume-b9869d42-418f-4d7c-b4bf-951b035d1817
[root@host2 ~]# tgt-admin –delete iqn.2010-10.org.openstack:volume-b9869d42-418f-4d7c-b4bf-951b035d1817

mysql> use cinder;
mysql> update volumes set status = ‘error’, attach_status = ‘detached’ where id = ‘b9869d42-418f-4d7c-b4bf-951b035d1817′;
Query OK, 1 row affected (0.04 sec)
Rows matched: 1 Changed: 1 Warnings: 0

[root@host2 ~]# cinder delete b9869d42-418f-4d7c-b4bf-951b035d1817

Now that the volume is cleaned up the instance needs to be massaged a bit too so that it can be torn down as well:

[root@host2 ~]# nova reset-state 4365e90f-b7cf-4253-9ded-1844df1c786b
[root@host2 ~]# nova delete 4365e90f-b7cf-4253-9ded-1844df1c786b

And if the instance still doesn’t want to delete set it to deleted in the db:

mysql> UPDATE instances SET  vm_state=’deleted’,task_state=NULL,deleted=1,deleted_at=now()  WHERE uuid=’4365e90f-b7cf-4253-9ded-1844df1c786b’;

If you had to edit the DB like this then go make sure that the instance is actually undefined on the compute node. For me I was able to look at the dashboard’s admin panel to see how many instances were on the compute node and make sure that virsh list reported the same number, and it did.

We use an internal ca and kerb at work. I got a new laptop that I’m setting up from a fresh install and trying not to copy settings over. Adding these to chrome isn’t something I do that often so figrured I’d doc for myself and help others find how to do these too.

Thanks for documenting this Anderson!

# install nsscert package
sudo yum install nss-tools

# import into nssdb
certutil -d sql:$HOME/.pki/nssdb -A -t TC -n “cert_name” -i cacert.crt

# if the above errors out because of authentication, set a password to nssdb
certutil -d sql:$HOME/.pki/nssdb -W # and then try again.

# add domain kerb auth to list permanently
vim /etc/opt/chrome/policies/managed/example.json
{ “AuthServerWhitelist”: “*.example.com”,
“AuthNegotiateDelegateWhitelist”: “*.example.com”
}

I’ve notice over the years of having this blog that I’ve collected a bunch of bogus registered users. I’ve put in some measures to reduce the number of them, but that only reduced the frequency of how often they showed up, it didn’t clean out the old ones. I finally sat down and did something about cleaning out the old ones. This isn’t setup to happen automatically, I just ran it manually and probably will continue to do it manually every once in a blue moon.

Easiest way for me to clean up users is to clean out those that haven’t commented and did not register in the past 24 hours. I ran this sql and it took care of the cruft.

DELETE FROM wp_usermeta WHERE user_id IN (
    SELECT * FROM (
        select wp_users.ID from wp_users
        left join wp_comments on wp_users.ID = wp_comments.user_id
        where DATEDIFF(curdate(), user_registered) > 1
        group by wp_users.ID
        having count(wp_comments.user_id) = 0
    ) AS p
);
DELETE FROM wp_users WHERE ID IN (
    SELECT * FROM (
        select wp_users.ID from wp_users
        left join wp_comments on wp_users.ID = wp_comments.user_id
        where DATEDIFF(curdate(), user_registered) > 1
        group by wp_users.ID
        having count(wp_comments.user_id) = 0
    ) AS p
);

I had to remove the networking specifics from a qcow2 openstack image to reimport into glance today.

There were lots of posts on how to use qemu-nbd, and all of them  had a comments that said “or you could use libguestfs”, but very few showed how to use it. So here’s what I did, in case you’re looking for a quick example.

I wanted to modify my RHEL image. When I tried to use guestmount  I wasn’t sure what the -m meant at first. Guestmount was nice enough to let me know what it wanted there and what my options were. LVM was picked up and supported no problem.

[root@radez ~]$ guestmount -a RHEL-6.4-x86_64.qcow2 -m /dev/sdx1 /mnt
libguestfs: error: mount_options: mount_options_stub: /dev/sdx1: No such file or directory
guestmount: ‘/dev/sdx1′ could not be mounted.
guestmount: Did you mean to mount one of these filesystems?
guestmount: /dev/sda1 (ext4)
guestmount: /dev/VolGroup00/LogVol00 (ext4)
guestmount: /dev/VolGroup00/LogVol01 (swap)
[root@radez ~]$ guestmount -a RHEL-6.4-x86_64.qcow2 -m /dev/VolGroup00/LogVol00 /mnt

now remove the network stuff that shouldn’t be in this image

[root@radez ~]$ cd /mnt
[root@radez mnt]$ vim etc/sysconfig/network-scripts/ifcfg-eth0 # remove HWADDR and UUID
[root@radez mnt]$ rm -f etc/udev/rules.d/70-persistent-net.rules
[root@radez mnt]$ rm -rf etc/ssh/ssh_host*

lastly cleanup, and load the image up into glance

[root@radez mnt]$ cd
[root@radez ~]$ umount /mnt
[root@radez ~]$ glance image-create –name RHEL-6.4-x86_64 –is-public 1 –disk-format qcow2 –container-format bare –file RHEL-6.4-x86_64.qcow2

Since my last post on thermal quite a bit of development has happened, and a new screencast was posted. This all happened in December 2012. I’ve since not been able to work on thermal due to a new role on the openstack team. Though, I wanted to post an update so that the most current information is available.

As of the screencast below thermal includes the following features:

• core heat features converted to python-heatclient
• catalogues feature
• unit tests
• concept of template designer

Screencast: http://fedorapeople.org/~radez/thermal20121205.ogv

Since this screencast Heat has been incubated (congrats heat team!), which has also migrated all the code repos out of the heat-api project on github. I’ve posted the new link below to the thermal repo in Steve Baker’s github account.

There’s also been discussion with Horizon that when Heat becomes core (I think in the Havanna release) that thermal will most likely be merged into the dashboard code base.

Links:
https://github.com/steveb/heat-horizon
http://wiki.openstack.org/Heat
http://jsplumb.org/jquery/demo.html

I was recently contacted  asking if I would be willing to review and share a graphic regarding “The Rise of Android”. Graphic has some really interesting information about android. I enjoyed reading through it.

I’m often asked which phone is better? People are usually referring to iPhone vs Android. My cookie cutter answer goes like this: “It depends”.

I’ve owned my Nexus 4 for a few months now and I love it. My personal choice when I buy is Android.
Though, both have strengths, both have weaknesses. Each individual person has their own set of requirements for a phone. Each individual person has their own preferences about a phone. Pick the OS that you’re comfortable with that meets your needs, then find a piece of hardware that’s a good deal and is up to date.

In the end whether you buy a Nexus 4, Galaxy SIII, iPhone 5 or an iPhone 4 you’re going to get a solid piece of hardware that can meet most of your needs. (**there are other android phones on the market right now too)

I’ve been working on a thermal, a webui for heat.
Heat is a recently incubated component into openstack that uses templates to launch instances to deploy application stacks into openstack.

Here’s a recent screencast of what it looks like.

I’ve gotten to a point that I needed to establish some test coverage. Each of the panels has a tests.py file so I figured that those tests should be picked up and run with the base openstack_dashboard run_tests.sh. Turns out they can with a few tweaks.

1. install your dashboard into openstack_daskboard/tests/settings.py.

dradez@tirreno:~/git/horizon git:(master*)➤ git diff openstack_dashboard/test/settings.py
diff --git a/openstack_dashboard/test/settings.py b/openstack_dashboard/test/settings.py
index ddbb38c..b034285 100644
--- a/openstack_dashboard/test/settings.py
+++ b/openstack_dashboard/test/settings.py
@@ -35,6 +35,7 @@ INSTALLED_APPS = (
     'openstack_dashboard.dashboards.project',
     'openstack_dashboard.dashboards.admin',
     'openstack_dashboard.dashboards.settings',
+    'thermal',
 )

AUTHENTICATION_BACKENDS = ('openstack_auth.backend.KeystoneBackend',)
@@ -42,7 +43,7 @@ AUTHENTICATION_BACKENDS = ('openstack_auth.backend.KeystoneBackend',)
SITE_BRANDING = 'OpenStack'

HORIZON_CONFIG = {
-    'dashboards': ('project', 'admin', 'settings'),
+    'dashboards': ('project', 'admin', 'settings', 'thermal'),
     'default_dashboard': 'project',
     "password_validator": {
         "regex": '^.{8,18}$',

2. add your dashboard to the run_tests_all command

dradez@tirreno:~/git/horizon git:(master*)➤ git diff run_tests.sh
diff --git a/run_tests.sh b/run_tests.sh
index d3357c1..587d8a4 100755
--- a/run_tests.sh
+++ b/run_tests.sh
@@ -302,7 +302,7 @@ function run_tests_all {
   if [ "$NOSE_WITH_HTML_OUTPUT" = '1' ]; then
     export NOSE_HTML_OUT_FILE='dashboard_nose_results.html'
   fi
-  ${command_wrapper} coverage run -p $root/manage.py test openstack_dashboard --settings=openstack_dashboard.test.settings $testopts
+  ${command_wrapper} coverage run -p $root/manage.py test openstack_dashboard --settings=openstack_dashboard.test.settings thermal $testopts
   # get results of the openstack_dashboard tests
   DASHBOARD_RESULT=$?

3. run ./run_tests.sh

This is just the way that I got it to work that I’ve used to establish my initial unit test coverage. I’m sure there are other ways to do this. There’s probably a simple patch that would allow run_tests.sh to handle this without having to add in your dashboard name. Maybe it already can and I just haven’t come across it.

just FYI: I used this page to get started with building my dashboard.

**** Update****
I got curious what it would take to not hard code your dashboard name into the run_tests.sh file, turns out it was a simple patch there too:

dradez@tirreno:~/workspace/git/horizon git:(master*)➤ git diff run_tests.sh
diff --git a/run_tests.sh b/run_tests.sh
index d3357c1..8ae5901 100755
--- a/run_tests.sh
+++ b/run_tests.sh
@@ -282,7 +282,7 @@ function run_tests {
 }

 function run_tests_subset {
-  project=`echo $testargs | awk -F. '{print $1}'`
+  project=`echo $testargs | awk '{print $1}' | awk -F. '{print $1}'`
   ${command_wrapper} python $root/manage.py test --settings=$project.test.settings $testopts $testargs
 }

then you can run:

dradez@tirreno:~/git/horizon git:(master*)➤ ./run_tests.sh openstack_dashboard thermal

This is actually preferable for me as it skips the horizon tests, which I don’t need to run each time.
- You can’t just pass thermal as it doesn’t have a settings.py file.
- You can’t pass openstack_dashboard.thermal because thermal is not a submodule of openstack_dashboard.

Two upgrades today:

1. Fedora 17 -> Fedora 18 Beta
Started to use Preupgrade but it’s been depricated and it pointed me to just use yum instead.
Flawless upgrade, Great work on F18 Fedora Community!

Also looking forward to the Preupgrade replacement tool that’s mentioned on the Preupgrade page.

2. Galaxy SII -> Nexus 4
What a beautiful piece of hardware. Nice work Google / LG!
I had to get a new sim card because the Nexus 4 uses a micro-sim. Tried to go to an AT&T authorized re-seller to get my phone put on the network and they tried to charge me $29.99 for a new sim. WHAT?! I walked out and visited a AT&T corporate store later that day and got my new sim for FREE!

Next day I got a txt msg saying I had used over $50 in pre-paid data. WHAT?!
I’m not a pre-paid account so I called up and asked what was going on. Turns out my 2G data package got taken off somewhere in the process of getting my new sim and never put back on. So I was being charge $2 a Mb, and had used 44.27 Mb since the previous evening. Yes, at  this point I was bracing for a $88+ charge on my next bill for data.

Kudos to AT&T,
They reestablished my data package then because they seemingly couldn’t just erase the prepaid data charges they gave me a $131.99 credit on my account. So I made about $40 off of their mistake.

Thanks for taking good care of me AT&T!

Thermal is an openstack horizon web ui dashboard/panel that exposes heat functionality.

Some of the links in this post have been updated or moved.
Please see the updated post: http://www.jaddog.org/2013/03/18/thermal-update/

Openstack: http://www.openstack.org/
Horizon: http://wiki.openstack.org/Horizon/
Heat: http://heat-api.org/
Thermal: https://github.com/heat-api/heat-horizon
Screencast: http://radez.fedorapeople.org/thermal20121024.ogv

There’s still a long way to go, this is simply to show proof of concept work integrated into horizon.

Current Features:
- upload a stack template, fill out its parameters and launch it
- delete a stack
- ajax status updating on a stack’s progress.

Planned features
- stack details and events
- load a template from a url
- template cataloging
- Heat HA support

Heat is currently in process of moving to launchpad, until then I don’t plan to generate wiki docs. I’m going to keep basic installation information updated in the README file for now.

Also, though none exist as of this writing, I’m planning to generate some github issues to keep up with what’s being done. File bugs and such there for now.

—–BEGIN CERTIFICATE—–
MII…snip…
—–END CERTIFICATE—–

Had fun working on an keystone bug this morning:
https://bugs.launchpad.net/keystone/+bug/1060389

Resulted in this change:
https://review.openstack.org/14309

But in the process we figured out why a PKI cert starts with MII by reading the ANS.1 spec.
Explained the reason in a comment in the patch committed, here’s the answer to $title without the keystone justification included:

thx to ayoung for sorting this out.

base64 decoded hex representation of MII is 3082
In [3]: binascii.hexlify(base64.b64decode(‘MII=’))
Out[3]: ’3082′

according to: http://www.itu.int/ITU-T/studygroups/com17/languages/X.690-0207.pdf
pg4: For tags from 0 to 30 the first octet is the identfier
pg10: Hex 30 means sequence, followed by the length of that sequence.
pg5: Second octet is the length octet
first bit indicates short or long form, next 7 bits encode the number
of subsequent octets that make up the content length octets as an
unsigned binary int

82 = 10000010 (first bit indicates long form)
0000010 = 2 octets (next 7 bits indicate octet count for content length)
so read the next 2 octets to get the length of the content.