The Evil Triangle: El Capitan, SUS and Caching

The Evil Triangle: El Capitan, SUS and Caching

Update: As of December 8th, 2015, this issue is now resolved.
In order to have this fully corrected you must have:
– El Capitan 10.11.2 for clients
– Server 5.0.15 for Caching Server

Since the El Capitan betas, there has been a significant issue with software updates. Unfortunately as of 10.11.1, this issue is still not resolved and as I no longer have a resolution in sight, I figured it would be best to publicly document the issue.

In order to see this issue, you must have the following configuration:
– Apple SUS or Reposado
– Caching Server
– El Capitan client pointing to your SUS server.

Technical Overview

During the betas, it was quite clear that Apple was adding more client features to the Caching Service. Apple announced Personal iCloud Data and Apple Configurator 2, which finally had support for Caching Servers. One unannounced feature though was a change to softwareupdated.

Let’s look at how everything falls apart

El Capitan 10.11.0

10.11.0 Client Machine Terminal

softwareupdate -d -v -a
Software Update Tool
Copyright 2002-2015 Apple Inc.

Finding available software

Downloading Example Package

10.11.0 Client Machine install.log

softwareupdated: softwareupdated: Catalog URL changed (from "https://swscan.apple.com/content/catalogs/others/index-10.11-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.gz" to "http://example.sus.com/index.sucatalog"). Resetting state.
softwareupdated: SUScan: Using catalog http://example.sus.com/index.sucatalog
softwareupdated: 1 update found:
zzzzexample | Example Package
softwareupdated: ContentLocator: Modified URL is: http://example.caching.server:port/example.pkg?source=example.sus.com
softwareupdated: Finished downloading package example.pkg to file:///var/folders/zz/zzyx/C/com.apple.SoftwareUpdate/CFNetworkDownload.tmp (error (null)) from peer: example.caching.server
softwareupdated: zzzzexample: Failed post-download size check for package "example.pkg": expected 101010101, got 0

As you can see, the following takes place:
1. softwareupdate is ran and noticed that either com.apple.SoftwareUpdate.plist has been modified or a profile has been applied to re-configure the Software Update Catalog.
2. A sub process SUScan begins to look for updates and finds 1 available update.
3. A sub process called ContentLocator (client name for Caching Service, not to be confused with ContentLocatorService) finds an available caching server and redirects softwareupdate to use it.
4. softwareupdate attempts to download the package, but for some undetermined reason it fails (more on this soon) and downloads an invalid .pkg file with a 0kb file size.
5. At this point, softwareupdate is completely confused and hangs indefinitely until its process is killed.

Caching Server 5.0.4 and below

Caching Server Debug.log

Request from example.client:port [Software Update (unknown version) CFNetwork/760.1.2 Darwin/15.0.0 (x86_64)] for http://example.caching.server:port/example.pkg?source=example.sus.com denied because the source is not whitelisted.

From the looks of it, Apple has a hardcoded list of domains that are whitelisted and all other domains simply fail to register. As the internal SUS is not on this list, it does not trust it.

Apple’s Bandaid fix for 10.11.1 and Server 5.0.15

Today Apple released 10.11.1 and Server 5.0.15. I immediately updated and began testing. While some work has been done, the overall issue still remains.

10.11.1 Client Machine Terminal

softwareupdate -d -v -a
Software Update Tool
Copyright 2002-2015 Apple Inc.

Finding available software

Downloading Example Package
Error downloading Example Package Update: The network connection was lost.
Done.

Error downloading updates.

10.11.1 Client Machine install.log

softwareupdated: softwareupdated: Catalog URL changed (from "https://swscan.apple.com/content/catalogs/others/index-10.11-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.gz" to "http://example.sus.com/index.sucatalog"). Resetting state.
softwareupdated: SUScan: Using catalog http://example.sus.com/index.sucatalog
softwareupdated: 1 update found:
zzzzexample | Example Package
softwareupdated: ContentLocator: Modified URL is: http://example.caching.server:port/example.pkg?source=example.sus.com
softwareupdated: No more tasks - invalidating session now
softwareupdated: SoftwareUpdate: error download of zzzzexmaple: Error Domain=NSURLErrorDomain Code=-1005 "The network connection was lost." UserInfo={NSUnderlyingError=0x7fe12f0ade10 {Error Domain=kCFErrorDomainCFNetwork Code=-1005 "(null)" UserInfo={_kCFStreamErrorCodeKey=-4, _kCFStreamErrorDomainKey=4}}, NSErrorFailingURLStringKey=http://example.caching.server:port/example.pkg?source=example.sus.com, NSErrorFailingURLKey=http://example.caching.server:port/example.pkg?source=example.sus.com, _kCFStreamErrorDomainKey=4, _kCFStreamErrorCodeKey=-4, NSLocalizedDescription=The network connection was lost.}
softwareupdated: Stopping transaction with ID [0x3]
softwareupdated: SoftwareUpdate: Removed foreground transaction [0x3]

So what’s changed?

Well it looks like Apple is simply severing the connection and preventing the hang. While it fixes some issues, the major issue still remains: Clients cannot receive updates.

Caching Server 5.0.15

Caching Server Debug.log

Request from example.client:port [Software Update (unknown version) CFNetwork/760.1.2 Darwin/15.0.0 (x86_64)] for http://example.caching.server:port/example.pkg?source=example.sus.com aborted because the source is not whitelisted.

As you can see, while the domain is still not whitelisted, it sends an abort rather than a deny and the client has been updated to understand the difference.

Awesome fix Apple!

On top of this, I discovered the Darwin version was not updated for the second year a row. At least they are consistent at being inconsistent.

Houston, we have a lot of problems

Below is a running list of items that fail to work and the configurations that cause them.

JAMF Casper customers may also be impacted by this issue, but to date I have not worked with another administrator that is using this combination.

El Capitan 10.11.0 Client / Caching Server 5.0.4 and lower / Apple SUS or Reposado

• Mac App Store Updates
• Examples:
• iTunes
• Document Camera Raw Compatibility
• 10.11.1 Delta Update
• 10.11.2 Combo Update
• Printer Driver Installations
• Legacy Java Installations
• Xcode Command Line Tools installations
• Example: typing git clone in Terminal
• Munki
• Bootstrapping and attempting to download Apple updates such as iTunes. Munki will indefinitely hang until the bootstrapping process is terminated.
• If Munki is ran via --auto mode (Launch Daemon or manual) and discovers an Apple update, Munki will indefinitely hang. The user will be unable to use Managed Software Center and all subsequent attempts to run --auto will also fail until all hung processes are killed or the machine is rebooted.

El Capitan 10.11.1 Client / Caching Server 5.0.4 and lower / Apple SUS or Reposado

Sadly, with this configuration there are 0 issues resolved. All 10.11.0 client issues remain.

El Capitan 10.11.1 / Caching Server 5.0.15 / Apple SUS or Reposado

• Mac App Store Updates
• Examples:
• iTunes
• Document Camera Raw Compatibility
• 10.11.2 Delta Update
• Printer Driver Installations
• Legacy Java Installations
• Xcode Command Line Tools installations
• Example: typing git clone in Terminal
• Munki
• Bootstrapping and attempting to download Apple updates such as iTunes. Munki will indefinitely loop until the bootstrapping process is terminated.

Workarounds

While there are several workarounds available, I don’t find any of them particularly ideal. Each organization should weigh the pros/cons and make a determination as to the best course of action.

All of these workarounds will work instantly for the Mac App Store and softwareupdated, however if Munki is hung, all munki processes will need to be killed or the machine will need to be rebooted.

Disable Caching Server

• Pros
• Easiest solution as it requires no reconfiguration of client machines
• Resolves all documented issues
• Cons
• Increased bandwidth for iOS/OS X devices

Point El Capitan clients to Apple’s Software Update Servers

• Pros
• Resolves all documented issues
• Cons
• Loss of apple software update scheduling
• Increased bandwidth for OS X devices
• In some cases, this may not work
• Example of Apple’s own server being blocked: http://example.caching.server:port/example.pkg?source=osxapps.itunes.apple.com denied because the source is not whitelisted.

Do not approve El Capitan Software Updates

• Pros
• Fixes Munki Bootstrap issues
• Easy with Reposado
• Cons
• Apple SUS users may lose updates for other Operating Systems
• Items such as Java, Xcode CLI and Printer Drivers will fail to install for users.

Downgrade to Yosemite

• Pros
• Resolves all documented issues
• Does not require disabling other enterprise products
• Cons
• Cannot downgrade October 2015 iMac refresh
• Somewhat less secure than El Capitan (SIP, Kernel protections)

Thankfully, Apple has also issued Security Update 2015-007 which should resolve some (but not all) of the security issues with Yosemite.

So now what?

Unfortunately, I no longer have an ETA for when this will be resolved. All signs point to 10.11.2, but without a current beta to test, I cannot confirm that the behavior is resolved.

If you are running into this issue and have a Sales Engineer, please reach out to them with impact data. The more pressure we can place on Apple, the sooner this issue can be put to rest.

Paradise Island – Hiding El Capitan’s Free Upgrade Banner

Yesterday, El Capitan came out and with it came great fanfare…except if you aren’t ready to deploy it.

Maybe there is a Microsoft bug holding you back or perhaps it’s even an Apple One. Regardless, when your users now open the App Store update tab, they will see this.

Lovely.

Let’s dive a little deeper…

WebKit

The App Store itself uses WebKit/HTML to display most of its content. Looking at the source file of the Update tab, you will see a div inserted specifically for El Capitan.

<div id="utd-os-updates" class="lockup-container installations utd-updates updates hidden">
    <table class="no-header">
        <tr class="'installation'">
          <td>
            <div class="artwork"></div>
          </td>
          <td class="description">
            <h2>OS X El Capitan</h2>
            <p class="tagline">A refined experience and improved performance for your Mac.</p>
            <p class="blurb">The next big release of the world's most advanced desktop operating system. Now available as a free upgrade.</p>
            <a href='https://itunes.apple.com/us/app/id1018109117?mt=12' class="learn-more">Learn More</a>
          </td>
          <td>
              <span class="update">
                <span class="status"></span>
                <button class="install-button hidden"></button>
              </span>
              <div class="multi-button install-button update-button">

<span class="price">Free Upgrade</span>
<span class="left-cap"></span>
<div class="inner"><span>Free Upgrade</span></div>
<span class="right-cap"></span>
</button>

</div>
          </td>
        </tr>
    </table>
</div>

If you notice, there is a context-menu class that contains the phrase hideosupdate. This class is what allows you to disable the screen via the first method.

Method 1 – Right Click – Hide Update

Any user can simply right click on the banner and select Hide Update

Simple enough right? But what actually happens when you press this button?

The Island

When a user disables the banner, a SQLite database is created at the following location: ~/Library/Containers/com.apple.appstore/Data/Library/WebKit/LocalStorage/https_su.itunes.apple.com_0.localstorage

Upon opening this database, you will find a key called didHideUTDIsland. Apple refers to this banner as the Island and if you pay close attention, the original div also had a mention of “utd”. What exactly this means, I’m not 100% sure, but I would assume it simply stands for “update”

To test this, you can easily hide and unhide the banner by moving this database out it’s location and then moving it back in. Each time you will need to close out of the App Store or refresh the window.

Method 2 – Deploying file.

So now that we have the file, we must ensure that this database file is installed onto every user’s library. I’m a huge fan of Outset and more recently Munki-Pkg (you don’t need to be a munki user to use munki-pkg!), so this will be my preferred method of deployment.

Personally, when a script calls for a file, I place it in /usr/local/outset/custom. This folder does not exist in the default Outset installation, so my munki-pkg creates contains the logic to support this. With that said, it is still pretty straight forward.

The following Outset login-once script should be sufficient:

#!/bin/sh

if [ -e ~/Library/Containers/com.apple.appstore/Data/Library/WebKit/LocalStorage/https_su.itunes.apple.com_0.localstorage ]
    then
        rm -rf ~/Library/Containers/com.apple.appstore/Data/Library/WebKit/LocalStorage/https_su.itunes.apple.com_0.localstorage
        cp -R /usr/local/outset/custom/https_su.itunes.apple.com_0.localstorage ~/Library/Containers/com.apple.appstore/Data/Library/WebKit/LocalStorage/
    else
        mkdir -p ~/Library/Containers/com.apple.appstore/Data/Library/WebKit/LocalStorage
        cp -R /usr/local/outset/custom/https_su.itunes.apple.com_0.localstorage ~/Library/Containers/com.apple.appstore/Data/Library/WebKit/LocalStorage/
fi

By simply dropping a script into /usr/local/outset/login-once, Outset will cause every user that logs in to run it, in their own context.

You will find my the script I am using, as well as the munki-pkg files at my Github

Caveat Emptor

Since we are removing this file if it exists, there is a chance that users may lose other updates that they have hidden. Some may find this an added bonus, while others will shake their heads in disapproval.

If either of these items concern you, you may want to rewrite the script to not remove the file and simply check if it exists first.

Notes

If you have never used munki-pkg before, it is a very straight forward approach to building packages. There is no UI and it is fast and easy to learn. Hopefully this is reason enough to test it out.

Sorry Apple!

Cacher 2 – Dropping Server 4 Support and Future Plans.

Current State

If you’re reading this, chances are you know about Cacher. If you haven’t, please read my previous article detailing Cacher and how to set it up. You can find it here

With the release of Apple’s Server 5, iOS 9 and El Capitan, Cacher has been receiving some increased attention. To add to that, a few of my buddies have been tweeting about it (Thanks Ben and Arek)

Today I’d like to announce Cacher “2.0”

Here are some of the changes:
– Support for Server 5
– Support for large caching servers who transfer terabytes of content
– Add logic for cases where Cacher does not understand and request relevant logs
– Removed support for Server 4.x
– Better mathematical equations for calculating bandwidth statistics

The Future

When I first started writing this, it was only for internal use. It has gone from ~160 lines of bash to over 750 lines of terrible bashisms and comments. All of my new scripts are currently being written in Python and while bash is comfortable for me, Cacher is a perfect example of why I should be using Python.

Add to that, there are still a few issues I haven’t resolved with Cacher 1 or 2.

Allister Banks began work on Sashay a few months ago and has been slowly adding features to it. Our current plans are (time permitting) to merge projects and ensure that people using either tool are not impacted.

Enjoy Cacher 2 and be on the lookout for Cacher stats inside of Sashay.

Re-Introducing Cacher

Several months ago I was tasked to prove that our brand new Caching Server was actually doing it’s job. I had already setup CacheWarmer with an hourly custom script/cronjob, so I knew when new iOS builds would come out, we wouldn’t be crippled.

SNMP could technically be an option but the Caching Service relies on random high ports. We could make assumptions, but that still wouldn’t be good enough to justify why we had this installed in out data center. Anecdotally, I knew it was working: Our users noticed app store improvements, I noticed app store improvements, but our executives wanted metrics.

Apple’s Server information is very minimal and that’s being generous.

E-mailing screenshots filled with squiggly line isn’t good enough. Maybe the logs can help?

Reading the Advanced Caching Documentation page, I went ahead and enabled “LogClientIdentity” by running the following command.

sudo serveradmin settings caching:LogClientIdentity = 1

After looking at the generated log files, I knew I was on to something. The logs gave me everything I needed, but with roughly 30,000 machines possibly hitting our caching server, the logs were enormous. I needed to figure out something.

Enter Cacher

Cacher is a bash script that if configured as a daily cronjob, will e-mail you daily statistics on your devices.

Cacher requires the following:
– OS X Yosemite 10.10.3
– Server 4.1 with Alerts Enabled

Currently there are two branches: Master and HTML.

Master -> Outputs a standard message to Server 4’s alert mechanism.

HTML -> Adds some html to the outputted message to trick Server 4’s alert mechanism into sending indented text.

Here is an example e-mail alert from the HTML branch.

Installation

Installation is pretty simple:

1. Download Cacher from either branch (I personally use the HTML branch).
2. Place it somewhere on your OS X Caching Server.
3. Setup a cronjob/LaunchD

Example

Create a folder and download it

mkdir -p /Users/Shared/Cacher
curl "https://raw.githubusercontent.com/erikng/Cacher/HTML/Cacher" -o "/Users/Shared/Cacher/Cacher"

Make sure it can be executed

chmod a+x /Users/Shared/Cacher/Cacher

Create a root cronjob. This is required for the alert mechanism to work. Alternatively you could create a LaunchDaemon as well, but I find this easier.

sudo env EDITOR=nano crontab -e

Nano will open. Put in your preferred e-mail time alert. I personally have an e-mail sent at 6:30 AM every day.

30      6       *       *       *       /Users/Shared/Cacher/Cacher

After entering this into nano, use hit CTRL+X and save it. If successful, crontab will give you an installation status. If you need more information on setting up a different time, please see this maclife article

Now sit back, wait until tomorrow to see your new statistics and wonder why Apple doesn’t have something like already.

Moving your DeployStudio Workflows to Imagr

If you’re here, chances are you’re interested in Imagr and read a blog post or two or three. Unfortunately, something is holding you back. Maybe it’s time, your imaging process or a very specific function of DeployStudio that you absolutely need. As you’ll quickly find out, most barriers are short lived.

Benefits of Imagr:

• Open Source
• Light
• You actually know the maintainers!

Areas of Improvement:

• Server Logging
• Embedded Workflows
• Fusion Drive Support

Imagr isn’t:

• A tool for capturing thick images
• GUI based for the Admin

---

A few basic scripts

I’ve created two Imagr Wiki pages:
– DeployStudio Alternative Scripts
– Admin Provided Scripts

Sir Gilbert has opted for the Wiki route for a few reasons:
– Smaller codebase to maintain
– Easier entry for admins to contribute to the project.

Currently there are only a few scripts. As you begin to transition over to Imagr, if there is a configuration setting that you find to script, please add it to the list.

---

Breaking down your DeployStudio Workflows

I’ve taken DeployStudio for granted for many years. While I document many other processes, due to DeployStudio being mostly WYSIWYG, I’ve never felt compelled to actually list out each process I used.

If you want a successful transition you’re going to want to document. You know those checkboxes you use in DeployStudio? Document them!

Deploy Studio Breakdown Example

1. DS Restore Task
- Restore System Recovery Partition
- Set as Default Startup Volume
- Preventative Volume Repair
- Convert to CoreStorage
2. DS HostName Task
3. DS Configure Task
- Skip Apple Setup Assistant
- Disable Gatekeeper
4. DS Generic Task
- Munki Manifest Selector
5. DS Package Install Task
- Munki
6. DS SoftwareUpdate Task
7. Time Task
8. DS Active Directory Task
9. Automatic Reboot after completion (built into DS)

Let’s tackle these one by one. All of these tasks should be added to the components array

DS Restore Task

This one can is rather simple. By default, Imagr will automatically bless the volume.

<dict>
  <key>type</key>
  <string>image</string>
  <key>url</key>
  <string>http://10.10.10.10/imagr/masters/OS_X_10.10.3-14D136.hfs.dmg</string>
</dict>
<dict>
  <key>type</key>
  <string>script</string>
  <key>content</key>
  <string>#!/bin/bash
# Repair Disk Permissions
diskutil repairPermissions /
  </string>
</dict>
<dict>
  <key>type</key>
  <string>script</string>
  <key>content</key>
  <string>#!/bin/bash
# Convert to CoreStorage
diskutil cs convert disk0s2
  </string>
</dict>

DS HostName Task

Imagr can now prompt for a name.

<dict>
    <key>type</key>
    <string>computer_name</string>
</dict>

If your naming convention is based via serial number (like me) you can even remove your custom DS script.

<dict>
    <key>type</key>
    <string>computer_name</string>
    <key>use_serial</key>
    <true/>
    <key>auto</key>
    <true/>
</dict>

DS Configure Task

Both of these tasks are rather simple.

<dict>
  <key>type</key>
  <string>script</string>
  <key>content</key>
  <string>#!/bin/bash
# Disable Gatekeeper
spctl --master-disable
  </string>
</dict>
<dict>
  <key>type</key>
  <string>script</string>
  <key>content</key>
  <string>#!/bin/bash
# Bypass Apple Assistant
/usr/bin/touch "{{target_volume}}/private/var/db/.AppleSetupDone"
  </string>
</dict>

DS Generic Task

Basically everything we are doing here are considered “Generic Tasks”. See my other post for an approach to non-scripted generic tasks.

DS Package Install Task

Packages are very straight forward

<dict>
    <key>type</key>
    <string>package</string>
    <key>url</key>
    <string>http://10.10.10.10/imagr/packages/munkitools.pkg</string>
    <key>first_boot</key>
    <false/>
</dict>

DS SoftwareUpdate Task

Here is where I recommend a mobile configuration file. If you still want to do it the DeployStudio way, here is an example.

<dict>
  <key>type</key>
  <string>script</string>
  <key>content</key>
  <string>#!/bin/sh
# Variables
SUS="URLPATH"

/usr/bin/defaults write "{{target_volume}}/Library/Preferences/com.apple.SoftwareUpdate" CatalogURL $SUS
chmod 644 "{{target_volume}}/Library/Preferences/com.apple.SoftwareUpdate.plist"
/usr/sbin/chown root:admin "{{target_volume}}/Library/Preferences/com.apple.SoftwareUpdate.plist"
  </string>
</dict>

Time Task

Rich Trouton has a great script to accomplish this.

<dict>
  <key>type</key>
  <string>script</string>
  <key>content</key>
  <string>#!/bin/sh
#Primary Time server for Company Macs

TimeServer1=timeserver1.company.com

#Secondary Time server for Company Macs

TimeServer2=timeserver2.company.com

#Tertiary Time Server for Company Macs, used outside of Company network

TimeServer3=time.apple.com

# Time zone for Company Macs

TimeZone=America/New_York

# Configure network time server and region

# Set the time zone
/usr/sbin/systemsetup -settimezone $TimeZone

# Set the primary network server with systemsetup -setnetworktimeserver
# Using this command will clear /etc/ntp.conf of existing entries and
# add the primary time server as the first line.

/usr/sbin/systemsetup -setnetworktimeserver $TimeServer1

# Add the secondary time server as the second line in /etc/ntp.conf
echo "server $TimeServer2" >> /etc/ntp.conf

# Add the tertiary time server as the third line in /etc/ntp.conf
echo "server $TimeServer3" >> /etc/ntp.conf

# Enables the Mac to set its clock using the network time server(s)
/usr/sbin/systemsetup -setusingnetworktime on
</string>
</dict>

DS Active Directory Task

I would highly recommend that you package this as saving this directly in the imagr_config.plist will leave your AD binding account exposed.

With that said, Sir Gilbert has a great script for this (taken from DS.)

<dict>
  <key>type</key>
  <string>script</string>
  <key>content</key>
  <string>#!/bin/sh

# This was stolen from DeployStudio. I didn't write it, but dammit, I'm going to use it.

#
# Script config
#

AD_DOMAIN="ad.company.com"
COMPUTER_ID=`/usr/sbin/scutil --get LocalHostName`
COMPUTERS_OU="OU=Macs,OU=London,DC=ad,DC=company,DC=com"
ADMIN_LOGIN="bindUser"
ADMIN_PWD="bindPassword"
MOBILE="enable"
MOBILE_CONFIRM="disable"
LOCAL_HOME="enable"
USE_UNC_PATHS="enable"
UNC_PATHS_PROTOCOL="smb"
PACKET_SIGN="allow"
PACKET_ENCRYPT="allow"
PASSWORD_INTERVAL="0"
ADMIN_GROUPS="COMPANY\Domain Admins,COMPANY\Enterprise Admins"

# UID_MAPPING=
# GID_MAPPING=
# GGID_MAPPING==

# disable history characters
histchars=

SCRIPT_NAME=`basename "${0}"`

echo "${SCRIPT_NAME} - v1.26 ("`date`")"

#
# functions
#
is_ip_address() {
  IP_REGEX="\b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b"
  IP_CHECK=`echo ${1} | egrep ${IP_REGEX}`
  if [ ${#IP_CHECK} -gt 0 ]
  then
    return 0
  else
    return 1
  fi
}


#
# Wait for the naming script to have run
#
if [ ${COMPUTER_ID} -eq "" ]
then
echo "The mac doesn't have a name, exiting."
  exit 1
fi

# AD can only use a 15 character name
COMPUTER_ID=`echo ${COMPUTER_ID} | cut -c1-15`

#
# Wait for network services to be initialized
#
echo "Checking for the default route to be active..."
ATTEMPTS=0
MAX_ATTEMPTS=18
while ! (netstat -rn -f inet | grep -q default)
do
  if [ ${ATTEMPTS} -le ${MAX_ATTEMPTS} ]
  then
    echo "Waiting for the default route to be active..."
    sleep 10
    ATTEMPTS=`expr ${ATTEMPTS} + 1`
  else
    echo "Network not configured, AD binding failed (${MAX_ATTEMPTS} attempts), will retry at next boot!" 2>&1
    exit 1
  fi
done

#
# Wait for the related server to be reachable
# NB: AD service entries must be correctly set in DNS
#
SUCCESS=
is_ip_address "${AD_DOMAIN}"
if [ ${?} -eq 0 ]
then
  # the AD_DOMAIN variable contains an IP address, let's try to ping the server
  echo "Testing ${AD_DOMAIN} reachability" 2>&1
  if ping -t 5 -c 1 "${AD_DOMAIN}" | grep "round-trip"
  then
    echo "Ping successful!" 2>&1
    SUCCESS="YES"
  else
    echo "Ping failed..." 2>&1
  fi
else
  ATTEMPTS=0
  MAX_ATTEMPTS=12
  while [ -z "${SUCCESS}" ]
  do
    if [ ${ATTEMPTS} -lt ${MAX_ATTEMPTS} ]
    then
      AD_DOMAIN_IPS=( `host "${AD_DOMAIN}" | grep " has address " | cut -f 4 -d " "` )
      for AD_DOMAIN_IP in ${AD_DOMAIN_IPS[@]}
      do
        echo "Testing ${AD_DOMAIN} reachability on address ${AD_DOMAIN_IP}" 2>&1
        if ping -t 5 -c 1 ${AD_DOMAIN_IP} | grep "round-trip"
        then
          echo "Ping successful!" 2>&1
          SUCCESS="YES"
        else
          echo "Ping failed..." 2>&1
        fi
        if [ "${SUCCESS}" = "YES" ]
        then
          break
        fi
      done
      if [ -z "${SUCCESS}" ]
      then
        echo "An error occurred while trying to get ${AD_DOMAIN} IP addresses, new attempt in 10 seconds..." 2>&1
        sleep 10
        ATTEMPTS=`expr ${ATTEMPTS} + 1`
      fi
    else
      echo "Cannot get any IP address for ${AD_DOMAIN} (${MAX_ATTEMPTS} attempts), aborting lookup..." 2>&1
      break
    fi
  done
fi

if [ -z "${SUCCESS}" ]
then
  echo "Cannot reach any IP address of the domain ${AD_DOMAIN}." 2>&1
  echo "AD binding failed, will retry at next boot!" 2>&1
  exit 1
fi

#
# Unbinding computer first
#
echo "Unbinding computer..." 2>&1
dsconfigad -remove -username "${ADMIN_LOGIN}" -password "${ADMIN_PWD}" 2>&1

#
# Try to bind the computer
#
ATTEMPTS=0
MAX_ATTEMPTS=12
SUCCESS=
while [ -z "${SUCCESS}" ]
do
  if [ ${ATTEMPTS} -le ${MAX_ATTEMPTS} ]
  then
    echo "Binding computer to domain ${AD_DOMAIN}..." 2>&1
    dsconfigad -add "${AD_DOMAIN}" -computer "${COMPUTER_ID}" -ou "${COMPUTERS_OU}" -username "${ADMIN_LOGIN}" -password "${ADMIN_PWD}" -force 2>&1
    IS_BOUND=`dsconfigad -show | grep "Active Directory Domain"`
    if [ -n "${IS_BOUND}" ]
    then
      SUCCESS="YES"
    else
      echo "An error occured while trying to bind this computer to AD, new attempt in 10 seconds..." 2>&1
      sleep 10
      ATTEMPTS=`expr ${ATTEMPTS} + 1`
    fi
  else
    echo "AD binding failed (${MAX_ATTEMPTS} attempts), will retry at next boot!" 2>&1
    SUCCESS="NO"
  fi
done

if [ "${SUCCESS}" = "YES" ]
then
  #
  # Update AD plugin options
  #
  echo "Setting AD plugin options..." 2>&1
  dsconfigad -mobile ${MOBILE} 2>&1
  sleep 1
  dsconfigad -mobileconfirm ${MOBILE_CONFIRM} 2>&1
  sleep 1
  dsconfigad -localhome ${LOCAL_HOME} 2>&1
  sleep 1
  dsconfigad -useuncpath ${USE_UNC_PATHS} 2>&1
  sleep 1
  dsconfigad -protocol ${UNC_PATHS_PROTOCOL} 2>&1
  sleep 1
  dsconfigad -packetsign ${PACKET_SIGN} 2>&1
  sleep 1
  dsconfigad -packetencrypt ${PACKET_ENCRYPT} 2>&1
  sleep 1
  dsconfigad -passinterval ${PASSWORD_INTERVAL} 2>&1
  if [ -n "${ADMIN_GROUPS}" ]
  then
    sleep 1
    dsconfigad -groups "${ADMIN_GROUPS}" 2>&1
  fi
  sleep 1

  if [ -n "${AUTH_DOMAIN}" ] && [ "${AUTH_DOMAIN}" != 'All Domains' ]
  then
    dsconfigad -alldomains disable 2>&1
  else
    dsconfigad -alldomains enable 2>&1
  fi
  AD_SEARCH_PATH=`dscl /Search -read / CSPSearchPath | grep "Active Directory" | sed 's/^ *//' | sed 's/ *$//'`
  if [ -n "${AD_SEARCH_PATH}" ]
  then
    echo "Deleting '${AD_SEARCH_PATH}' from authentication search path..." 2>&1
    dscl localhost -delete /Search CSPSearchPath "${AD_SEARCH_PATH}" 2>/dev/null
    echo "Deleting '${AD_SEARCH_PATH}' from contacts search path..." 2>&1
    dscl localhost -delete /Contact CSPSearchPath "${AD_SEARCH_PATH}" 2>/dev/null
  fi
  dscl localhost -create /Search SearchPolicy CSPSearchPath 2>&1
  dscl localhost -create /Contact SearchPolicy CSPSearchPath 2>&1
  AD_DOMAIN_NODE=`dscl localhost -list "/Active Directory" | head -n 1`
  if [ "${AD_DOMAIN_NODE}" = "All Domains" ]
  then
    AD_SEARCH_PATH="/Active Directory/All Domains"
  elif [ -n "${AUTH_DOMAIN}" ] && [ "${AUTH_DOMAIN}" != 'All Domains' ]
  then
    AD_SEARCH_PATH="/Active Directory/${AD_DOMAIN_NODE}/${AUTH_DOMAIN}"
  else
    AD_SEARCH_PATH="/Active Directory/${AD_DOMAIN_NODE}/All Domains"
  fi
  echo "Adding '${AD_SEARCH_PATH}' to authentication search path..." 2>&1
  dscl localhost -append /Search CSPSearchPath "${AD_SEARCH_PATH}"
  echo "Adding '${AD_SEARCH_PATH}' to contacts search path..." 2>&1
  dscl localhost -append /Contact CSPSearchPath "${AD_SEARCH_PATH}"

  if [ -n "${UID_MAPPING}" ]
  then
    sleep 1
    dsconfigad -uid "${UID_MAPPING}" 2>&1
  fi
  if [ -n "${GID_MAPPING}" ]
  then
    sleep 1
    dsconfigad -gid "${GID_MAPPING}" 2>&1
  fi
  if [ -n "${GGID_MAPPING}" ]
  then
    sleep 1
    dsconfigad -ggid "${GGID_MAPPING}" 2>&1
  fi

  GROUP_MEMBERS=`dscl /Local/Default -read /Groups/com.apple.access_loginwindow GroupMembers 2>/dev/null`
  NESTED_GROUPS=`dscl /Local/Default -read /Groups/com.apple.access_loginwindow NestedGroups 2>/dev/null`
  if [ -z "${GROUP_MEMBERS}" ] && [ -z "${NESTED_GROUPS}" ]
  then
    echo "Enabling network users login..." 2>&1
    dseditgroup -o edit -n /Local/Default -a netaccounts -t group com.apple.access_loginwindow 2>/dev/null
  fi

  #
  # Self-removal
  #
  if [ "${SUCCESS}" = "YES" ]
  then
    if [ -e "/System/Library/CoreServices/ServerVersion.plist" ]
    then
      DEFAULT_REALM=`more /Library/Preferences/edu.mit.Kerberos | grep default_realm | awk '{ print $3 }'`
      if [ -n "${DEFAULT_REALM}" ]
      then
        echo "The binding process looks good, will try to configure Kerberized services on this machine for the default realm ${DEFAULT_REALM}..." 2>&1
        /usr/sbin/sso_util configure -r "${DEFAULT_REALM}" -a "${ADMIN_LOGIN}" -p "${ADMIN_PWD}" all
      fi
      #
      # Give OD a chance to fully apply new settings
      #
      echo "Applying changes..." 2>&1
      sleep 10
    fi
    if [ -e "${CONFIG_FILE}" ]
    then
      /usr/bin/srm -mf "${CONFIG_FILE}"
    fi
    /usr/bin/srm -mf "${0}"
    exit 0
  fi
fi

exit 1
</string>
</dict>

DS Automatic Reboot

If you would like Image to automatically reboot after deploying your image (and begin it’s first boot processes), add this somewhere in your workflow.

<key>restart_action</key>
<string>restart</string>

Final Thoughts

As you can see, many of DeployStudio’s checkboxes are very small configuration changes. While none of these examples contain any error checking, they work quite well.

Through the years my workflows have slimmed down considerably. Most have been converted to mobile configuration files, while others have simply moved into munki/outset. While you may consider some aspects of DeployStudio essential, I continue to find myself decreasing the amount of steps needed for a “successful image”. As you transfer your workflows over to Imagr, think about trimming down as much fat as possible. In the end your results will be much more stable and you’ll become a more dynamic admin.

Using Munki Manifest Selector with Imagr

There is one tool that is integral to my technicians workflow and my sanity. With several hundred locations, I needed a way to properly set Munki’s ClientIdentifier. At previous employers, I was able to define a machine naming convention that worked well, but with so many moving parts where I am at now, I needed something dynamic, consistent and stable: Munki Manifest Selector

If you have never used it, I highly recommend at least trying it out. It requires minimal work and although I use an internal fork, the functionality is for the most part identical. Joe has a great post here

As I began to test and move my workflows to Imagr, I hit a road block. While both DeployStudio and Imagr have a generic task, they are radically different architecturally.

DeployStudio vs Imagr Generic Task:

DeployStudio

Since DeployStudio mounts a network volume, the Runtime has direct access to MMS. All an admin needs to do is place both Munki Manifest Selector.app and a corresponding script into the DeployStudio Scripts folder and then call it with a Generic Task. Here is an example script.

#!/bin/bash
BASE_DIR=`dirname "${0}"`
$BASE_DIR/Munki\ Manifest\ Selector.app/Contents/MacOS/Munki\ Manifest\ Selector\
    --targetVolume "${DS_LAST_SELECTED_TARGET}"\

Imagr

Imagr’s Generic Tasks are completely different. In many ways they mimic the behavior of Munki. Scripts are taken from imagr_config.plist and then ran.

<dict>
    <key>type</key>
    <string>script</string>
    <key>content</key>
    <string>#!/bin/bash
/usr/bin/touch "{{target_volume}}/some_file"</string>
    <key>first_boot</key>
    <false/>
</dict>

Unlike DeployStudio, all Imagr components are downloaded individually. What’s an admin to do?

Get Creative – Curl to the rescue!

To begin, let’s download MMS here. For this example please rename the dmg to “Munki_Manifest_Selector.dmg”

If you’ve followed Nick McSpadden’s Imagr Guide, in your recently made Imagr folder, create a new folder called “packages” and place the DMG there.

mkdir -p /yourmunkirepo/imagr/packages

After placing it there, we need to add a workflow to your imagr_config.plist but let’s break this down first.

After pulling down the image and verifying a successful deployment, we are going to utilize Imagr’s Generic Task with first_boot disabled. This task is going to do a few things:
– Curl the DMG (using the new deployment’s curl binary)
– Mount the dmg using hdiutil from the NBI
– Run Munki Manifest Selector and wait for user input
– Unmount the DMG using hdiutil from the NBI
– Remove the DMG from the deployment

Currently, AutoNBI does not add the curl binary. Don’t fret though – it is going to happen. For now we will use this somewhat hacky method.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>workflows</key>
  <array>
    <dict>
      <key>name</key>
      <string>Munki Manifest Selector Test </string>
      <key>restart_action</key>
      <string>restart</string>
      <key>description</key>
      <string>This workflow deploys an AutoDMG base image with Munki Manifest Selector</string>
      <key>components</key>
      <array>
        <dict>
          <key>type</key>
          <string>image</string>
          <key>url</key>
          <string>http://10.10.10.10/imagr/masters/OS_X_10.10.3-14D136.hfs.dmg</string>
        </dict>
        <dict>
            <key>type</key>
            <string>script</string>
            <key>content</key>
            <string>#!/bin/bash

# Downloading MMS DMG
"/Volumes/Macintosh HD/usr/bin/curl" http://10.10.10.10/imagr/packages/Munki_Manifest_Selector.dmg -o "/Volumes/Macintosh HD/private/tmp/Munki_Manifest_Selector.dmg"
sleep 1

# Mount MMS DMG
hdiutil attach "/Volumes/Macintosh HD/private/tmp/Munki_Manifest_Selector.dmg"
sleep 1

# Run MMS
"/Volumes/Munki_Manifest_Selector/Munki Manifest Selector.app/Contents/MacOS/Munki Manifest Selector" --targetVolume "/Volumes/Macintosh HD"
sleep 2

# Unmount MMS DMG
hdiutil unmount "/Volumes/Munki Manifest Selector"
sleep 1

# Delete MMS DMG
rm -rf "/Volumes/Macintosh HD/private/tmp/Munki_Manifest_Selector.dmg"

exit 0
            </string>
            <key>first_boot</key>
            <false/>
        </dict>
  </array>
</dict>
</plist>

Voila!

That’s it. There’s no need to re-architect MMS – just simply wrap it in a DMG. Obviously this is a first version and there isn’t any download verification but with MMS being so small (~100k) for most deployments this should be sufficient.

Don’t stop yourself from moving to Imagr. Get creative and you will be happy with the results.

Yosemite Style Banners for Munki 2

Customizing munki can be quite rewarding when done right. By utilizing the client_resources.zip file you can make a great looking GUI for your users.

Trick 1: Add CSS into footer_template.html

Bart Reardon first documented this trick. You can modify (for better or worse) the css Munki utilizes by adding your desired changes directly into the footer_template.html. The following is what I am currently using to make Munki look more like the App Store in Yosemite.

<style>
html, body {
    -webkit-background-size: auto;
    background-repeat: repeat-x;
    background-color: #ffffff;
    background: -webkit-linear-gradient(top, #ffffff 0%, #f5f5f5 75%, #e1e1e1 100%);
    background-attachment: fixed;
}

This simply changes the background color to white and adds a subtle gradient to the bottom of the window. You can add CSS to any template file, but the footer_template is on every page view.

Trick 2: Adding an icon to the sidebar (sidebar_template.html)

Adding an icon to the sidebar is quite simple – simply add your .png file to your resources folder and use a relative link. Make sure you center it (and don’t go over 128×128 px) or it will look terrible!

<div class="sidebar">
    <div class="chart titled-box quick-links">
        <h2>Quick Links</h2>
        <div class="content">
           		<div class="artwork">
                	<center><img target="_blank" href="https://github.com/munki/munki" width="128" height="128" alt="Munki Github" class="artwork" src="custom/resources/MSC.png" />
                </div>
            </ol>
        </div>
    </div>
</div>

Trick 3: Linking banners to Optional Installs (showcase_template.html)

This isn’t really a trick, but more of a feature that isn’t completely documented. You can link individual banners to specific items in your repository.

Below you will find three examples: All Categories, Music and iMovie. Make sure you add .html to each item or it will not work.

<div class="showcase">
    <div class="stage" onClick='stageClicked();'>
        <img href="categories.html" alt="Categories" src="custom/resources/App_Store_1.png" />
        <img href="category-Music.html" alt="Music Category" src="custom/resources/Making_Music.png" />
        <img href="detail-iMovie.html" alt="iMovie" src="custom/resources/iMovie.png" />
    </div>
</div>

Erik’s Sweet Giveaway (Oprah Style)

Attached below are several banners that I have modified from the Mac App Store. These banners are the correct dimension (1158×200) and configured in a way so no matter how small MSC is, items will not be cut off. Enjoy!