Hosting multiple Phabricator instances on a single server (Ubuntu 14.04)

Preface:

  • Sorry for the WoT, it’s kind of complex (for me)
  • What you need to do is a mixture of the advanced and the “normal” install guide, and a bit of guesswork
  • Phacility probably does not endorse this installation style, but I hope it will work and survive

 

Sources of information for this topic:

 

Prerequisites:

  • Ubuntu 14.04 LTS installed and updated on phserver
  • MySQL >= 5.5 running on mysqlserver
  • have SSL certificate and corresponding key for phab1.your.domain ready (in your homedir ~)

 

Assumptions:

  • new Phabricator instance will be called phab1
  • MySQL is not on the same machine
  • You’re logged in to phserver as a user with sudo privileges

 

Work steps:

  • install “standard” software (if new server)
  • create an instance directory
  • create an apache site file
  • paste the following content
  • install certificates by copying them to their destinations
  • install Phabricator components
  • this will have created three directories below /var/www/phab1/: arcanist, libphutil and phabricator
  • enable the apache site
  • restart apache
  • create a mysql user phab1 on mysqlserver
  • grant that user full access to all of his (not yet existing) databases
  • working on the console, you must have an environment variable indicating the instance you want to work with
  • create /var/www/phab1/phabricator/conf/custom/phab1.conf.php (no closing php tag is intentional)

  • set the storage namespace for this instance, so that database names start with phab1_. The -E parameter is important for passing environment variables to sudo.
  • the file /var/www/phab1/phabricator/conf/local/local.json should now look like this:
  • make the instance directory owned by the web server
  • install Phabricator databases and schemas
  • this will create all needed databases, and apply patches to them directly. Reply to questions with “y”
  • now visit https://phab1.your.domain/ with your browser and create an admin user
  • after that, Phabricator will show the Setup issues page (/config/issue/) with all tasks that need to be completed before you’re finally ready.
  • one of the tasks says that “Phabricator daemons are not running”. Create /etc/init/phab1-phd.conf:
  • create daemon working directories
  • set Phabricator daemon working directories
  • start Phabricator daemons

After these steps you should have a working instance of Phabricator, that works independantly from other instances on that same machine, except for using the same web server. What I cannot say at the moment, is whether all Phabricator modules will work smoothly this way. I hope they all respect the environament variables and/or the custom settings. One of the most vital things for working with this setup, especially when setting config values on the console, is to use the -E parameter. The path from where scripts or binaries (phd f.e.) are started also plays a role.

We went this way because we didn’t want to throw a new VM at each instance. A different way of addressing this could be to put Phabricator in a container, but this is beyond my scope currently. I hope this will turn out to be a working solution. Comments welcome.

Getting the server response when Invoke-Webrequest results in error 5xx

Hi there,

lately, I’ve been confronted with a problem where I simply couldn’t get meaningful error messages from failing runs of Invoke-Webrequest. In doing something like this:

$err would only ever contain the error code like “500 bla”, but not what the server actually returned as the error message. I was querying a REST API.

A colleague finally found the answer here.

What needs to be done is, to tell Powershell to read the stream. The exact difference I cannot tell, as this happens only for 5xx errors, as far as I can see.

$response will now contain the body of what the remote web service returned.

This is another good case that shows me, how easily one can get lost doing g**gle searches. I had searched for a solution also, but the sheer mass of results has overwhelmed me. Also, using only slightly different search terms always give totally different search results… Maybe I was using the wrong terms… Maybe I should have also read the documentation for Invoke-Webrequest more closely 🙂

Testing IMAP4 + TLS + AUTH with cygwin (or linux), Thunderbird debug

From time to time I have to do some troubleshooting for the IMAP4 protocol on our Exchange infrastructure. Most of the clients are running Thunderbird. For this purpose you can do two things.

    • Start Thunderbird in debugging mode

As described in this wiki article, create a batch with the following content:

This will generate imap.log on your desktop, which you can monitor while Thunderbird is talking to the mail server.

    • Connect to IMAP on command line

You will need a linux box or have cygwin installed to do this. IMAP4 is listening on port 143.

You will see the certificate exchange passing by, and you will have a blinking cursor. From here you can do IMAP console commands, as described here or here, and elsewhere 🙂
For example, to log in as user123 with password foobar, do this: (important: do not mistype, you cannot backspace and correct!)

Server’s response, in case credentials are correct, will be

You can now let the server show you all available folders:

Server’s reply, as an example:

So, INBOX has 8 subfolders. Select one of the folders.

Server’s reply, with some details about the folder:

Let’s fetch one of the emails:

This will give you the whole of the first email, including headers, ending with

You’ve seen enough, so log out:

Server’s last words (for now 🙂

Testing SMTP + TLS + AUTH communication with cygwin or a linux machine

Ever felt the need to see what a mail server actually does when another mail server or a mail client (Thunderbird f.e.) connects to it? Easy to trace when the connection isn’t encrypted – but rarely happens today. So you will have to “play” mail client yourself from the command line, which turns out to be a bit tricky. An excellent article on the topic can be found here.

Quick summary:

You will either need access to a linux box or have cygwin installed – or have openssl windows binaries, but the Base64 stuff will be hard 😉 (Go ahead, just install cygwin, a serious admin cannot live without…!)

Step 1: create the authentication bits

Step2: connect to the mail server through openssl

Step3: after saying HELO or EHLO to mailserver, you have to authenticate

The part after the “PLAIN” is the token that was returned in Step 1.

If you get back some sort of 200+ return code from the mailserver, you can start babbling SMTP as usual.

Use Powershell and 7-zip to zip up files in a directory individually

Today I needed to put the files I had in a directory into zip files, each one in its own archive. After fiddling around with how to quote and double quote stuff in PoSh I found this link on the technet forums. See Section 5 there.

This is how it’s done:

$arg1…$arg4 are the arguments to 7zip. “a” means add, “-mx3” means fast compression (still a whole lot more effective than zip), $arg2 is the destination archive, $arg3 is the file to be zipped.

Hope this helps.

Private Cloud Project – Ch02 – The Design Concept

This is a follow-up to my first post in this series, Private Cloud Project – Ch01 – The Mission.

As the mission was now set, I had to find resources how to go forward with the components that would make up our private cloud:

  • Hyper-V cluster
  • File Server cluster
  • Scale Out file server
  • Storage Spaces

When I started gathering information about these components, especially those related to storage, I first had to learn the nomenclature or vocabulary that was used in this area. “Storage Spaces”, f.e., which has been introduced in Windows Server 2012, is fairly new to the tech world, and usable resources, except for “marketing stuff”, are hard to find. To this date (September 2014) only few people seem to be using Storage Spaces, or actively writing about it, although it’s a great technology from Microsoft. A few gadgets are still missing, associated to management, which we all know from “standard” RAID controllers like the HP Smart and PERC. For example, you do not see the rebuild status for a drive when it has faulted and been replaced.

Also pretty hard to find was a hardware vendor whose whole stack of Server->Controller->Shelf->Drives would be suitable for Storage Spaces and also supported by Microsoft. After fiddling with Intel shelves connected to some DELL servers, talking to DELL representatives and well known local MVP, we settled with a design consisting of purely DELL equipment. A few of the certifications were still to be done by DELL, but as the process was obviously already advancing, we decided to go that way.

The result should look like this:
Microsoft private cloud stack
(Source)

The components of the environment are:

  1. The Hypervisor machines, based on Microsoft Hyper-V
  2. A central storage, based on Microsoft Windows Server 2012 R2 “Scale out File Server” (SOFS)
  3. • SOFS provides storage

  4. Microsoft System Center Virtual Machine Manager (VMM)
  5. • VM creation and management, workload deployment and distribution

  6. Microsoft System Center Data Protection Manager (DPM)
  7. • Backup

  8. Microsoft System Center Orchestrator (ORC)
  9. • Process automation of creation, deployment and monitoring

  10. Microsoft System Center App Controller (APC)
  11. • Self Service platform for internal customers

    As a more modern and more flexible alternative to using ORC+APC, Microsoft has issued the free “Azure Pack”, which is a port of their commercial platform software adapted for the usage with On-Premise private cloud environments.

  12. Azure Pack (AzP)
  13. • Self Service platform for admins and internal customers

Hypervisor Nodes

Hypervisors are the work horses of the Private Cloud environment. They host the Virtual Machines by virtualizing their physical hardware resources to the guest operating systems. To be able to host a significant number of virtual machines, the Hypervisors need to have powerful processor hardware, a large amount of memory and very fast network interfaces. The Hypervisor hardware setup consists of:

  • DELL Poweredge R620 (1U)
  • 2x E5-2650 v2
  • 16x 16GB DIMM (256GB)
  • NDC 4x I350 1GBit Ethernet
  • 2x X520 SFP LP PCIe Dual 10Gbit Network Card
  • 2x 300GB 10k SAS for System RAID1

Hypervisors have

  • Microsoft Windows Server 2012 R2 Datacenter installed
  • The Hyper-V role activated

Scale Out File Server (SOFS)

The SOFS replaces traditional enterprise storage systems. The SOFS will contain the virtual disk container files (VHDs) where the VMs which are hosted on the Hypervisor nodes store their data. The storage is made available to the Hypervisors through SMB 3.0, the known protocol for accessing file shares in the Windows world, which has been optimised for serving applications like MSSQL or – like in our case – VMs.

The SOFS is basically built on DELL PE 720xd, but contains some extras. These are:

  • Three DELL SAS Controllers (without RAID functionality)
  • An Intel X520 10Gbit network card
  • JBOD Hard Disk Shelves
  • SAS SSD Drives for storing “hot data”
  • SAS Hard Disks
  • 64GB of RAM

SOFS will be built as a cluster, with building blocks of initially two physical servers in a cluster. Both servers will be attached to three JBOD shelves, each exporting all of their hard drives to both servers. The SAS devices are managed by Windows Storage Spaces and are exported to the Hypervisors through the SOFS role as special “file shares”, f.e. “\SOFSvmshare”

The SOFS nodes will have

  • Microsoft Windows Server 2012 R2 Standard installed
  • The Failover Clustering feature installed
  • SOFS activated

Switch Fabric

Our network department advised us to some really fancy but also pretty expensive network hardware for setting up our environment. The 10 gigabit connections for the building block we were buidling would be provided by 2 Cisco N5K-C5596UP-FA switches. The 1 gigabit connections would be provided by 2 Cisco N2K-C2248TP-1GE, so-called “fabric extenders”, connected to the 10gb switches. They are switches that only have minimal logic of their own, the “real work” is done by the 10gb switch.

Here’s how hypervisors and file server are connected:

hy-net

We decided against converging the network connections on the hypervisor, as we would lose other functionality, like RSS. Instead we would use the first two onboard NICs (1gb) to build the “Managament Team”, then have a team of two of the 10gb ports – each one from a different 10gb NIC – for VM traffic, and the remaining two 10gb ports for storage traffic to the file servers via SMB3.

sofs-net

We used the same scheme for the file servers, just no VM team here.

All connections are distributed among the switches for redundancy. The teams are all LACP teams, therefore the switches need to be in a virtual chassis mode, so that LACP teams can be spread across the two physical units. Don’t ask for details, though, I do not know them 🙂

The rack space we intended to use would be set up like this:
dell-racks

So much for part 2 of the series.
In the next part: The management cluster (for the System Center machines).

Private Cloud Project – Ch01 – The Mission

This is the first post in a series that describes the private cloud project at the company I work with, from the idea to the release, with as much technical content as possible.

I work as Windows system administrator in a team of 6-8 colleagues, and together we run what we call the Office IT Windows server infrastructure for our company, which consists of some 600 servers, including most of Microsoft’s products (AD, File, Print, DHCP, Exchange, Sharepoint, MSSQL etc.) and a lot of third party applications running on Windows servers. Due to the fact that the products the company we work with sells are IT based, we are merely one of many teams that are engaged in running IT systems.

The traditional end-to-end process of supplying our internal customers, who we call “Business Partners”, with the server machines they need, operating systems, the networking and other infrastructure, was until now a very stony road of opening tickets and orders in various systems, waiting for replies, and coordinating several involved teams. To make it short, the process could – and would – often take more than a handful of weeks to complete. Most Business Partners never explicitly expressed their discontent wiht this situation, but we were aware that we needed a solution for this nuisance.

To circumvent the week long logistics hassle, the way to go would have been our own environment, with our own datacenter, purchasing and networking teams, and setting up things ourselves. Not possible, though, due to the way our business runs.

In the end, virtualization was our only choice: setting up hardware would take place only in seldom cases, and deploying server systems as quick as possible after the business partner’s request, if possible from a self service or ordering web interface, i.e. from weeks to hours.

We would also make use of the other advantages a virtualized infrastructure could bring: quicker and easier patching, no more need to keep records of hardware life cycles, raising the density and usage of hardware, lowering energy and rack space consumption immensely.

So, out I was sent into the intahwebz to search for solutions. As usual, price would play a role, but what was even more vital to a possible decision was manageability. Our team runs Windows systems and this is what we would have liked to stick with. In came the infamous known contenders: vmWare, Citrix, Xen etc al. What we didn’t like about most of them: their need for an enterprise storage partner. EMC, HP EVA, HITACHI, NetApp, whatever. In the end, we came to a solution that was rather new to the market, and still is: Hyper-V cluster nodes, using SMB3 shares provided by Windows Servers in a cluster, which make use of Windows Storage Spaces.

And here it is, the mission: Create a private cloud environment, consisting of the full MS stack, from top to bottom, migrate all vmWare (yes, we have that, too) VMs and most of the hardware based servers to it. (Cream on top: do something similar for VDI also, not part of this blog post series.)

The voyage we set out on and the challenges we encountered are what’s in the next posts:
Private Cloud Project – Ch02 – The Design Concept(unreleased)
Private Cloud Project – Ch03 – Management Cluster (unreleased)
Private Cloud Project – Ch04 – File Server Cluster (SOFS) (unreleased)
Private Cloud Project – Ch05 – Hyper-V cluster (unreleased)
Private Cloud Project – Ch06 – VM migration (V2V) (unreleased)
Private Cloud Project – Ch07 – Hardware migration (P2V)

Here are some links I used for getting some basic knowledge and/or ideas:

virtualizationmatrix
Hyper-V server blog Rachfahl IT solutions (de)
Thomas Maurer’s blog
Aidan Finn’s blog
Altaro’s HyperV blog
Keith Mayer’s blog (MS)
MS decision aid

Stay tuned, feedback is very welcome.
Cheers,
Peter.

Using dynamic VHDx for IO intensive VMs may not be a good idea

For the hasty reader:
Using dynamic VHDx for IO intensive workload will generate high CPU usage in the management OS, but only on the first three cores. Using Sysinternals Process Explorer we found out that there are exactly three threads in the “System” process (ID 4) called “vmbusr.sys” that are the root of the CPU usage. We researched RSS, VMQ and other things. Basically, the huge load went away when we changed from dynamic to fixed VHDx.

The longer story:
During the testing phase of our Private Cloud environment we also did IO tests using SQLIO in up to 30VMs on the hypervisor machines. The hypervisors talk SMB3 to the storage environment. We ran test with 8k IOs and 40-80k IOs. We always noticed that, as soon as the VMs started doing the heavy SQLIO based IO, in the management OS the cores 0 to 2 were under full fire:
20140626-01
Looking at the process tree in Process Explorer we found that the System process (which has ID 4) was showing this load in the Threads tab:
20140626-02

That didn’t really help much, as you will find very little information on the web about the Hyper-V VMBus system. One is the architectural description at MSDN. It says:

VMBus – Channel-based communication mechanism used for inter-partition communication and device enumeration on systems with multiple active virtualized partitions. The VMBus is installed with Hyper-V Integration Services.

Another one is Kristian Nese’s blog post, for Hyper-V in W2K8, but the basics should still be true.

Not very enlightening (pun intended) four our case; why should the VMs doing the SQLIO workload talk to each other….? Maybe device enumeration…? We tried to assess the problem from various angles, playing with the size of the SQLIO blocks, tuning SMB network interface paramters, VMQ settings (although these VMs weren’t doing guest network traffic). In a calm minute my colleague Christoph tried doing the SQLIO with a bunch of VMs that were slightly different than the others. Tada: normal CPU load distribution among all cores! The difference was easily found: the VHDs were of fixed size. We will yet have to find out if there’s a limit in the number of VMs running on a host for not showing the strange behaviour.

The bad news: this happens already with only 5 VMs. We have not done a full comparison test, but the high VMBus load also seems to be introducing a limit to the IO a VM with dynamic VHDx can do.

Any helpful comments, hints or tricks are highly welcome.

Cheers,
Peter.

Edit 2014-06-30: Post from a guy with a similar issue

Windows admin’s heaven: chocolatey.org

Recently, Microsoft issued Windows Management Framework v5 preview, which introduces a Powershell module called OneGet: “OneGet is a new way to discover and install software packages from around the web.”

OneGet itself in its first version introduces the repositories from chocolatey.org. To explain what Chocolatey is, the analogy to apt or apt-get in Debian linux is the easiest way. Need Notepad++, cygwin, Foxit reader, 7zip, Firefox, Sysinternals tools and many more on your machine(s)? Tired of visiting each and every website or fiddling with a bunch of update mechnanisms that sometimes work and sometimes don’t? There comes Chocolatey. Install all of them with a few keystrokes, and keep them up to date with even less. Made my day.

How to delete files older than x

Something I regularly need to do is search for files that are older than a certain amount of time and do something with them, like move or delete. Here’s a handy snippet how to do it in PoSh.

The question mark is an alias for Where-Object, -le means “less than or equal”, and AddMinutes has close relatives like AddDays… just use the autocomplete via the tab key after “Add” to get more.