IDL 8.2 released!

IDL 8.2 is now available for download from the Exelis VIS website. It includes many enhancements—primarily in (New) Graphics and in the IDL widget toolkit—that help strengthen IDL 8.

Here’s a list of some of the new/improved features in IDL 8.2, by category:

GraphicsThree examples of colorbars in IDL 8.2

  • COLORBAR: discrete, untethered, end caps
  • Erase method
  • Better thick line rendering
  • DejaVu TrueType fonts with Unicode characters
  • ARROW and SYMBOL are vectorized
  • MAPGRID labeling
  • LEGEND supports VECTOR plots
  • AXIS: untethered, label orientations

Widget toolkit

  • WIDGET_TREE: checkboxes, bitmap buttons, toggle state with space bar, set sensitivity of leaves
  • WIDGET_BUTTON: can use RGBA bitmaps, improved appearance on Windows
  • WIDGET_CONTROL: REDRAW keyword updates portions of a widget hierarchy

File access

  • BigTIFF support on all platforms
  • IDLffVideoWrite: can swap out VIS-supplied FFmpeg distro
  • OPeNDAP support through netCDF-4 library on Mac OS X and Linux

Language

  • JSON_PARSE and JSON_SERIALIZE functions
  • Where method for List and Hash
  • NULL keyword for ISA; null return from IDL_Container::Get

Workbench

  • Duplicate routine checking in IDL’s path
  • Drop-in plugin support
  • ENVI extension wizard
  • Updated text translations (de, es, fr, it, ja, ko, nl, pt_BR, ru, zh, zh_TW)
  • Updates to code templates, project build dialog
  • Limited content assist for Direct Graphics routines

Help system

  • Revamped image analysis section with new examples
  • Additional help on creating videos, swapping out FFmpeg library
  • Improved layout of help pages with many new examples

I’ll expand on this list and showcase what’s new in IDL 8.2 in a set of webinars on June 14. After you install IDL 8.2, please see the “What’s New in IDL 8.2” section linked from the main page of the IDL Help browser; it has a nice summary of the new/improved features with links to many examples.

Finally (and I’m not sure I should mention this, but it’s true), there are many helpful bugfixes: I counted 25+ bugs that directly affected my work that are now fixed.

In the coming weeks and months, I’ll show examples of the new and improved features in IDL 8.2, starting next week with a post about IDL’s native OPeNDAP support.

Update: Registration information for the What’s New in IDL 8.2 webinar is up on our Live Web Seminars page.

Posted in VIS | Tagged | 3 Comments

An ENVI 5 extension

Last week, I showed an example of an ENVI 5 batch program; this week, I’ll show an extension, the successor to a user function in ENVI Classic. (Be warned, though, that this turned into a long post.)

Extensions are programs executed in an interactive ENVI session. Extensions allow programmatic control over not only the file access and analysis components of ENVI, but also the UI. Like batch programs, extensions may use routines from both the ENVI and IDL libraries. ENVI+IDL is required to build an extension, but by compiling an extension into an IDL SAVE file, the extension can be executed in an ENVI (runtime version) session. I’ll outline this technique at the end of the post.

In this example, a user is prompted to select a file containing a raster image. The Prewitt edge enhancement operator is applied to each band of the image, with the result written to a new raster file. The first band of the input image is displayed in a new layer in the ENVI 5 UI, with the first band of the output image displayed in the same view as a portal.

Here’s the start of the extension code:

pro envi5_extension_ex
   compile_opt idl2

   catch, err
   if err ne 0 then begin
      catch, /cancel
      if e ne !null then $
         e.reporterror, 'Error: ' + !error_state.msg $
      else $
         message, !error_state.msg, /continue, /noname
      message, /reset
      return
   endif

   e = envi(/current)
   if e eq !null then $
      message, 'This extension requires an interactive ENVI session. Exiting.' 

I’ve chosen to write this extension as a procedure (a function or class would also be fine; note there’s an ENVI extension wizard in the IDL Workbench, located under the File menu, that sets up skeleton code for an extension). The CATCH block handles errors thrown in the program. Call the ENVI function to get the reference to the current ENVI session. If ENVI isn’t running, MESSAGE throws an error that’s handled by the CATCH block.

Next, prompt the user for a raster file to load:

   in_raster = e.ui.selectinputdata(/raster)
   if in_raster eq !null then return 

The SelectInputData method of the ENVIUI object presents a file picker dialog, the ENVI 5 analog to ENVI_SELECT in Classic. It returns an ENVIRaster object. Recall that an ENVIRaster is analogous to a FID in Classic, but it also holds metadata. If the user cancels the operation, then exit the extension.

Now prepare for the processed results:

   outdir = e.getpreference('output_directory')
   outfile = outdir + 'envi5_extension_ex.img'
   if file_test(outfile) then begin
      basenames = file_basename(outfile, '.img') + ['.img', '.img.enp', '.hdr']
      file_delete, outdir + basenames, /allow_nonexistent
   endif
   out_raster = e.createraster(outfile, inherits_from=in_raster)

In a manner similar to the batch program example from last week, I’ve set up a path to a file in the user’s output directory. The CreateRaster method makes a new ENVIRaster for the output. Because CreateRaster won’t allow an existing file to be overwritten, I’ve included some IDL code to delete the output file and its support files, if they exist. By using the INHERITS_FROM keyword, the CreateRaster method uses metadata from the input raster to populate the metadata for the output raster.

Now perform the processing. Iterate over the input bands, applying the Prewitt edge enhancement operator (with the IDL PREWITT function) to each and writing the result to the output file:

   widget_control, /hourglass
   for i=0, in_raster.nbands-1 do begin
      band = in_raster.getdata(bands=i)
      edge = bytscl(prewitt(band))
      out_raster.setdata, edge, bands=i
   endfor
   out_raster.save 

The GetData method is the ENVI 5 analog to ENVI_GET_DATA in Classic. The Save method is the analog to ENVI_WRITE_ENVI_FILE.

Finally, visualize the result in the ENVI UI. Using the default view, create new layers for the input and output images and display the first band of each, with the output displayed in a portal:

   view = e.getview()
   layer1 = view.createlayer(out_raster, bands=0, /clear_display)
   layer2 = view.createlayer(in_raster, bands=0)
   portal = view.createportal(layer=layer1, size=[150,150])
end 

As it currently stands, this extension could be executed in an interactive ENVI session with the Run button in the IDL Workbench, or by calling it from the IDL command prompt:

ENVI> envi5_extension_ex
% Compiled module: ENVI5_EXTENSION_EX.

However, analogous to a user function in Classic, an extension can be set up to automatically load into ENVI on startup and be visible in the Toolbox. To do this, two additional steps are needed.

Continue reading

Posted in data access, language, programming | Tagged , , , | 2 Comments

An ENVI 5 batch program

ENVI 5, released last week, has a new UI and API. I’d like to show a pair of examples that use the API: this week, a batch program; next week, an extension (the successor to a user function).

Batch programs are used to perform a sequence of processing tasks, typically noninteractively. They may use routines from both the ENVI and IDL libraries. Since batch programs are invoked from IDL, they require ENVI+IDL.

I’d like to show a familiar task: creating an NDVI image from the Bighorn Landsat 7 ETM+ scene included in the ENVI distribution. Here’s the beginning of the batch program:

pro envi5_batch_ex
   compile_opt idl2

   e = envi(/headless)

Calling ENVI as a function starts the interface to ENVI 5. (You can still access ENVI Classic (the new nickname for three-window ENVI) by calling ENVI as a procedure.) The HEADLESS keyword starts ENVI without a UI, similar to the Classic ENVI_BATCH_INIT routine. The return value from the ENVI function is a reference for this ENVI session.

Next, set up a path to the Bighorn Landsat file and get a reference to it:

   infile = filepath('bhtmref.img', $
      root_dir=e.root_dir, $
      subdirectory=['classic','data'])
   bh = e.openraster(infile)

The reference e is used here twice: the ROOT_DIR property holds the path to the ENVI installation directory; the OpenRaster method creates a new instance of the ENVIRaster class for this file. The reference bh is analogous to a FID in Classic, but it also holds image metadata, so an ENVIRaster in effect combines the functionality of ENVI_OPEN_FILE and ENVI_FILE_QUERY in Classic.

To compute NDVI, I’ll choose to use the Classic NDVI_DOIT routine. Most of the DOIT routines haven’t yet been ported from Classic to ENVI 5; this will occur over the next few service packs. I’ll show here a technique that’ll work in this transitional period.

NDVI_DOIT requires an input FID, which we don’t have. Use the ENVIRASTERTOFID function to extract a FID from the existing ENVIRaster object:

   bh_fid = envirastertofid(bh)

Note that ENVIRASTERTOFID has a complement, ENVIFIDTORASTER, that converts a FID to an ENVIRaster.

Next, set up additional information required by NDVI_DOIT:

   outfile = e.getpreference('output_directory') + 'bhtmref_ndvi.img'
   bh_dims = [-1, 0, bh.ncolumns-1, 0, bh.nrows-1]
   bh_pos = [4,3] - 1 ; Landsat [nir,red] bands
   bh_dt = 4 ; float

The GetPreference method allows programmatic access to certain ENVI preferences. I’m using it here to get the path to the user’s output directory, where NVDI_DOIT will write its result. The variables bh_dims, bh_pos and bh_dt set up the band dimensions, band position vector and data type for the NDVI calculation. Note how the number of samples and number of lines in the image are pulled from the ENVIRaster reference bh.

Now that setup is complete, call NDVI_DOIT through the ENVI_DOIT wrapper:

   envi_doit, 'ndvi_doit', $
      fid=bh_fid, $
      dims=bh_dims, $
      pos=bh_pos, $
      /check, $
      out_bname='Calculated NDVI', $
      out_dt=bh_dt, $
      out_name=outfile, $
      r_fid=r_fid
   print,'New file written to: ' + outfile

Finish the program by closing the headless ENVI session and returning to IDL:

   e.close
end

The Close method is the analogue to the Classic ENVI_BATCH_EXIT routine.

Execute this program with the Run button in the IDL Workbench or by calling it from the IDL command prompt:

IDL> envi5_batch_ex
% Compiled module: ENVI5_BATCH_EX.
% Restored file: ENVI.
New file written to: C:\mpiper\ENVI\output\bhtmref_ndvi.img

Grab the source code for this example here.

Posted in data access, language, programming | Tagged , , | 2 Comments

The merits of an example main program

When I write a new routine in IDL—a procedure or a function or a class—I like to include a main program at the bottom of the file. I use the main program to

  • demonstrate the calling syntax of the routine
  • give an example of how the routine is used
  • define a simple unit test (or tests)

I first saw this idea used in Python and I’ve copied it for my work in IDL.

For example, here’s the full code listing for a simple function, FLATTEN (which converts a multidimensional array into a vector) along with an example main. The programs are saved in the file flatten.pro, in a directory in IDL’s path.

; docformat = 'rst' 
;+ 
; A convenience routine that flattens/linearizes a 
; multidimensional array. 
; 
; :params: 
;  x : in, required, type=any array 
;   An array of any type to be converted to a vector. 
; 
; :author: 
;  Mark Piper, VIS, 2011 
;-
function flatten, x
   compile_opt idl2

   nx = n_elements(x)
   return, nx gt 0 ? reform(x, nx) : 0
end

; Example
a = indgen(5, 7)
b = flatten(a)
c = reform(a, n_elements(a))
help, a, b, c
print, 'Equivalent results? ', array_equal(b, c) ? 'Y' : 'N'
end 

By examining the main program, you can see how FLATTEN works; here, it’s used to convert a 5 x 7 array into a 35-element vector. To use the main program as an example, I execute it from the command line with the .run executive command:

IDL> .r flatten
% Compiled module: FLATTEN.
% Compiled module: $MAIN$.
A               INT       = Array[5, 7]
B               INT       = Array[35]
C               INT       = Array[35]
Equivalent results? Y

The .run command compiles both routines and executes the main program. We’d get the same behavior from the Run button (or the F8 keyboard shortcut) in the IDL Workbench.

Note that—and this is important—the calling mechanism still resolves the FLATTEN function (by itself) correctly:

IDL> .reset
IDL> x = indgen(2,3)
IDL> print, x
      0       1
      2       3
      4       5
IDL> y = flatten(x)
% Compiled module: FLATTEN.
IDL> print, y
      0       1       2       3       4       5

This means that (as intended) FLATTEN can be used as a library routine independent of its example main program.

I find this technique of including an example main to be especially useful with functions, which won’t execute with the Run button on the Workbench. (This may be a topic for another post, where I’d like to argue for an implicit redirect to !null for functions; e.g., FLATTEN could be called like this:

IDL> flatten(x)

without throwing a syntax error.)

Note: ENVI 5 was released this week. It has a new UI and a new API. The API still uses IDL, but with an object-oriented interface. Though I’m not a heavy ENVI user, I’d like to show some examples of using the new API over the next few weeks & months.

Posted in programming | Tagged , | 2 Comments

HDF and HDF-EOS Workshop XV

Last week I attended the HDF and HDF-EOS Workshop XV in Riverdale, MD. The purpose of this workshop is to discuss HDF products, tools and libraries with engineers from The HDF Group, scientists from NASA, NOAA and NSIDC, as well as vendors, like Exelis VIS, who provide software tools to work with HDF.

I thoroughly enjoyed my time at the workshop. (I think I’m turning into a file format wonk.) I was particularly interested in learning more about HDF5 and netCDF-4 integration (netCDF-4 is built on HDF5). I’ve used the classic netCDF data model (with dimensions, variables and attributes) for a long time. Now, though, there’s the option to use the HDF5-compliant “extended” data model with groups and datasets. (IDL supports both; this is good.) I think it’ll be interesting to see how scientists who are used to the classic model adapt to this change. Several talks addressed the topic of HDF5/netCDF-4 interoperability.

The complete workshop agenda is posted, along with abstracts and links to download each speaker’s slides. I gave a presentation on using IDL with Suomi NPP VIIRS data, similar to the ones I gave in March, but here I tried to emphasize how HDF5 made it very easy to access the data in a VIIRS file. My slides and example programs are available for download.

Posted in conference, data access, VIS | Tagged , | Leave a comment

An LSD radix sort algorithm in IDL

(Note: I’m at the HDF and HDF-EOS Workshop XV this week, so today I have a guest post by Atle Borsholm, a Senior Consultant in the Exelis VIS Professional Services Group. Atle is an IDL master. I’m trying to persuade him to share some of the cool programs he’s written on this blog. Here’s one.  –MP)

The IDL SORT function uses the quicksort (or qsort) algorithm. This is a memory-efficient sorting algorithm that handles any data type that supports comparison (e.g., operators like LT, GT, EQ). However, it can be slow for large input data. Wikipedia says the [proportional] speed of quicksort can range from n*log(n) to n^2 (where n is the number of elements). My main motivation for implementing a least significant digit (LSD) radix sort algorithm in IDL was to see if I could get a speed improvement, although a bonus with this algorithm is that it’s a so-called “stable” sort algorithm, meaning that the relative order of equal entries is retained. This is unfortunately not the case with IDL’s SORT. For example:

IDL> print, sort([2,2,3,2,3,3]), format='(6(i0,x))'
1 3 0 4 5 2

The first three indices in the result [1,3,0] all correspond to the number 2, which in a stable sort would be returned as [0,1,3]. The same goes for the 3s, which should be returned as [2,4,5]. So, a stable sort would give a result of [0,1,3,2,4,5] for this array.

An LSD radix sort is designed primarily for arrays of unsigned integer types of any length. The idea is to start sorting on the least significant digit of each array element. Since the digits have a fixed range, a histogram is used to place the entries into bins without regard for the other entries (except retaining the original order within each bin for stability). The algorithm continues by sorting on the next significant digit and so on until there are no more digits.

Here’s the first pass:

function radix_sort_r1, data, radix=radix
   compile_opt idl2, logical_predicate

   ; default radix if not specified
   if ~keyword_set(radix) then radix = 256
   radix = long64(radix)

   sorted = data
   mx = max(sorted)
   factor = 1ull
   repeat begin
      mx /= radix
      rem = sorted / factor
      digit = rem mod radix
      factor *= radix
      h = histogram(digit, min=0, max=radix-1, binsize=1, reverse_indices=ri)
      sorted = sorted[ri[radix+1:*]]
   endrep until mx eq 0

   return, sorted
end

Test RADIX_SORT_R1 on the array used above:

IDL> print, radix_sort_r1([2,2,3,2,3,3]), format='(6(i0,x))'
2 2 2 3 3 3

Sorting an array in-place is nice, but in most cases we want the list of indices — the way IDL’s SORT function returns its result — so we can use the indices on other arrays.

So, adding a few lines of code will also support the index as an output:

function radix_sort_r2, data, radix=radix, index=index
   compile_opt idl2, logical_predicate

   ; default radix if not specified
   if ~keyword_set(radix) then radix = 256
   radix = long64(radix)

   ; index output is requested
   if arg_present(index) then begin
      index = lindgen(n_elements(data))
   endif

   sorted = data
   mx = max(sorted)
   factor = 1ull
   repeat begin
      mx /= radix
      rem = sorted / factor
      digit = rem mod radix
      factor *= radix
      h = histogram(digit, min=0, max=radix-1, binsize=1, reverse_indices=ri)
      sorted = sorted[ri[radix+1:*]]
      if arg_present(index) then index = index[ri[radix+1:*]]
   endrep until mx eq 0

   return, sorted
end

Test on the array used above:

IDL> void = radix_sort_r2([2,2,3,2,3,3], index=index)
IDL> print, index, format='(6(i0,x))'
0 1 3 2 4 5

This is the result as expected from a stable sort algorithm.

Continue reading

Posted in data analysis, language, programming | Tagged , , , | 6 Comments

Using (New) Graphics in an IDL widget program, part II

Last week, I demonstrated how to set up a widget interface that embeds an IDL 8 (aka New) Graphics (NG) window with the WIDGET_WINDOW function. This week, let’s see how event handling works in this program. (Get the full program here.)

Start with the event handler for the slider widget:

pro widget_window_ex1_index_event, event
   compile_opt idl2

   ; Get the state variable.
   widget_control, event.top, get_uvalue=w
   state = w.uvalue

   ; Store the new image index and update the display.
   state['index'] = event.value
   widget_window_ex1_display, w
end 

This event handler follows the standard IDL widget programming model: it’s a procedure that accepts one parameter, the event structure, from XMANAGER.

Recall that I chose to store the NG window reference in the user value of the top-level base, with the actual state variable for the program in the user value of this NG window. This is nonstandard, but it gives one way to pass state information to both the widget event handler and the NG event handler.

To unpack the state variable, we reverse these steps in the event handler. In the last two statements, we get the value of the slider from the widget system and store it in the state variable, then call the helper _DISPLAY routine to refresh the window. We’ll look at this helper routine shortly, but let’s next review the event handler for mouse wheel events in the NG window:

function widget_window_ex1_mouse_wheel_event, w, x, y, d, k
   compile_opt idl2

   ; Get the state variable.
   state = w.uvalue

   ; Update the image index based on the value of "d". Also make
   ; sure the image index stays within [0, n_images-1].
   state['index'] += d
   state['index'] = (state['index'] > 0) < (state['n_images']-1)

   ; Update the display with the new image.
   widget_window_ex1_display, w

   ; Update the value of the slider.
   widget_control, state['index slider'], set_value=state['index']

   ; Cancel the default NG zoom behavior of the scroll wheel.
   return, 0
end 

Note that this NG event handler is a function. It accepts five parameters: w is the NG window reference, x and y give the event location in device coordinates, d gives the magnitude and direction (positive or negative) of the scroll wheel movement, while k stores the value of any modifier keys used. For more information on these parameters, as well as those for other types of NG window events, please see the entries for WIDGET_WINDOW and “Creating Mouse Event Functions” in the IDL Help.

We start this event handler by unpacking the state variable from the NG window reference. Next, we increment or decrement the index of the image by adding the value of the scroll wheel parameter d. This value is usually on [-1,1], but I recall once getting a value of -8 (try adding a PRINT statement to see the values of d). Ensure that the index is in the range [0,56] to avoid a subscripting error. The NG window can then be updated with the newly indexed image from the stack with the helper _DISPLAY routine. The slider widget also needs to be synced up with the new image index value. Last, NG event handlers are expected to return a value of 1, indicating that the default event handling action (for the scroll wheel, this is zooming) should be performed, or 0, which short-circuits the default handling and returns control to the standard widget event loop. I chose here to turn off the default zoom action.

Finally, the helper _DISPLAY routine:

pro widget_window_ex1_display, w
   compile_opt idl2

   state = w.uvalue
   w['head display'].setdata, (state['image'])[*,*,state['index']]
end 

This routine isn’t an event handler, but it helps in the event handling process. I chose to write it as a procedure and accept the NG window reference (which includes the state variable) as a parameter. Here’s where the NG NAME property is handy: the expression w['head display'] is the NG reference returned by IMAGE in the widget creation routine (where it’s called g). Here, the SetData method, introduced in IDL 8.1, is used to update the data used by IMAGE to the current image in the stack. The display in the NG window is automatically updated.

Here’s an animated GIF (made with IDL’s WRITE_GIF procedure) showing the program in action!

The widget program WIDGET_WINDOW_EX1 in action!

(I’ve been enjoying the recent resurgence of animated GIFs in simple scientific visualizations, like here – check out the IDL Direct Graphics!)

Posted in language, programming, visualization | Tagged , , , , , , | Leave a comment