Tutorial on how to use the MELT plugin

Audience: You are expected to be an advanced GCC user (fluent with command-line), and to have successfully downloaded and installed GCC MELT on your computer (probably under GNU/Linux running a recent version -4.7 at least- of GCC).

Notice that GCC MELT may use environment variables starting with GCCMELT or MELTGCC (e.g. GCCMELT_SOURCE_PATH or GCCMELT_MODULE_PATH etc, etc...). Don't use them unless you want to change the behavior of the GCC MELT plugin.

It is assumed that you have already installed a recent GCC MELT plugin on your Gnu/Linux system. See the download and build GCC MELT page for more. You may of course want to use gcc-4.7 or /usr/bin/gcc-4.7 or gcc.real (to bypass ccache on some systems) or whatver appropriate instead of gcc, and g++-4.7 instead of g++ etc, etc...

First steps in using the GCC MELT plugin.

The GCC MELT plugin is actually loaded, like every GCC plugin, by some compilation (by gcc, g++ etc...) of some source code. Since there is no way to run GCC MELT otherwise, it is sometimes convenient to have some empty source file, for example empty4melt.c (read "empty for MELT dot see"). You can create such an empty file with touch empty4melt.c command in a terminal (or later, in your Makefile, when relevant). The name of that empty file does not matter much, but it should have a .c extension (to be handled like a C file by gcc). That empty file could contain some comment, so you could also get it with e.g. date +"/* empty file at %c */" > empty4melt.c or likewise. If you are more used to C++ (or some other language supported by GCC) use an empty4melt.cc file for g++, etc. Of course when you use MELT on some of your own source files, you give them to gcc or g++ etc...

Compiling an empty file is often done for important GCC MELT side effects, such as translating a *.melt file (in MELT) to a *.so module. This is why it is so important that gcc -fplugin=melt don't go thru ccache etc.
Alternatively, instead of compiling some empty file, you could use the -x c option of GCC (specifying explicitly the C language) and compile the /dev/null file.

Invoking GCC with MELT

You should give, preferably as its first program argument, the -fplugin=melt option to gcc (or to g++, gfortran, gccgo etc....). If you pass only this argument, the GCC MELT plugin does nothing more. So running just the command  g++ -fplugin=melt -c yourexample.cc  has exactly the same effect as  g++ -c yourexample.cc  (except that the cc1plus compiler proper is dlopen-ing the GCC MELT plugin when you invoke the compiler as  g++ -fplugin=melt ).

Several parameters or settings of GCC MELT are wired inside the plugin. You can get the builtin settings with
gcc -fplugin=melt -fplugin-arg-melt-print-settings -c empty4melt.c -o /dev/null
which outputs on stdout the settings, in a shell-friendly format which could be source-d by /bin/sh (or any Posix friendly shell e.g. bash or zsh).

When really using GCC MELT to customize some GCC compilation, you'll want to give a work-directory with the -fplugin-arg-melt-workdir=work-directory program argument. This work-directory gets filled by GCC MELT with internal files (which may be re-used from one invocation to the next, so the work directory is somehow also a cache directory). The work directory is never cleared by MELT, and may grow indefinitely if you run MELT many times ; you may want to clear it from time to time (e.g. with your make clean or with a crontab entry).

The GCC MELT plugin does some real work only if given some mode with -fplugin-arg-melt-mode=mode. Without any mode, nothing special happens. Try to run
gcc -fplugin=melt -fplugin-arg-melt-mode=help -c empty4melt.c -o /dev/null
to get a list of available modes. Your MELT code usually should install [your] new modes.

Some interesting MELT modes

MELT modes for exploration of GCC internal representations

These exploring modes are new in MELT 1.0 and should interest a large number of GCC power users, even those not wishing to customize themselves GCC by coding their own MELT extensions.

The counting mode justcountipa (for just counting in inter-procedural analysis) can be used to give some simple statistics about Gimple instructions, Basic blocks, and emitted functions (after some inlining) by the GCC compiler. For example, to get such statistics when compiling the file general.c (with 1162 source lines) of GNU bash shell (version 4.2), just run:
make general.o CFLAGS='-O2 -fplugin=melt -fplugin-arg-melt-mode=justcountipa'
to get the following notices: 

general.c:1132:1: note: MELT inform: just counted 19 basic blocks and 39 gimple-s ↵
                  in function  <function_decl 0x7f7261dcdb00 get_group_array>
 get_group_array (ngp)
 ^
general.c:1099:1: note: MELT inform: just counted 19 basic blocks and 38 gimple-s ↵
                  in function  <function_decl 0x7f7261dcda00 get_group_list>
 get_group_list (ngp)
 ^
In file included from general.c:30:0:
/usr/include/unistd.h:715:12: note: MELT inform: just counted 16 basic blocks and 24 gimple-s ↵
                  in function  <function_decl 0x7f7261c6d700 group_member>
 extern int group_member (__gid_t __gid) __THROW;
            ^
general.c:1013:1: note: MELT inform: just counted 24 basic blocks and 85 gimple-s ↵
                  in function  <function_decl 0x7f7261a7be00 initialize_group_array>
 initialize_group_array ()
 ^
After more than 30 such notices, MELT enhanced GCC finally says 
general.c:65:1: note: MELT inform: just counted 7 basic blocks and 12 gimple-s ↵
      in function  <function_decl 0x7f7261dbd800 posix_initialize>
 posix_initialize (on)
 ^
In file included from /usr/include/features.h:371:0,
                 from /usr/include/wchar.h:27,
                 from config-bot.h:140,
                 from config.h:1129,
                 from general.c:21:
/usr/include/x86_64-linux-gnu/sys/stat.h:454:1: note: MELT inform: just ↵ 
       counted 3 basic blocks and 2 gimple-s ↵
       in function  <function_decl 0x7f7262016f00 stat>
 __NTH (stat (const char *__path, struct stat *__statbuf))
 ^
/usr/include/inttypes.h:331:1: note: MELT inform: just counted 3 basic blocks and 2 gimple-s ↵
      in function  <function_decl 0x7f7261fd8000 strtoimax>
 __NTH (strtoimax (const char *__restrict nptr, char **__restrict endptr,
 ^
cc1: note: MELT inform: just counted 41 functions, cumulating 517 basic blocks and 1046 gimples
Notice that the source locations are given from bottom of source file to top, because (strangely) inter-procedural analysis is done in that order by GCC. MELT has inserted its own melt_justcountipa simple inter-procedural analysis pass into GCC workflow, so you better give some optimization level to gcc. Also, function declarations are verbosely printed as GCC trees.

The findgimple mode (and also gofindgimple) is useful to find some gimples inside the internal representations of GCC. This is a quite powerful mode, and it requires two patterns (given in MELT syntax), the first πgimple being a pattern on Gimple-s, the second being a pattern πfun on function declaration Tree-s. It then shows within all functions of the compiled translated unit matching the πfun pattern all the gimples instructions matching the πgimple pattern. So our findgimple mode is a sort of super- grep working on internal GCC repreesentations of GCC; it does not use textual regexps but more "semantic" patterns. The simplest pattern is the catch-all wildcard (or joker) pattern ?_, it is always matching, so the match never fails. The wildcard is the default for both patterns. The gimple pattern πgimple is passed thru -fplugin-arg-melt-gimple-pattern=πgimple and the function pattern πfun is passed thru -fplugin-arg-melt-function-pattern=πfun. Optional action expressions (evaluated inside the match) can be passed with -fplugin-arg-melt-action=action and could use pattern variables bound in πgimple.

So to just find every gimple instruction in your translation unit, compile it with
-fplugin=melt -fplugin-arg-melt-mode=findgimple -fplugin-arg-melt-gimple-pattern='?_'
(we need to quote the argument for the shell because of the ? character, and we could even forget -fplugin-arg-melt-gimple-pattern='?_' since the wildcard is the default).

Such a compilation might be slow, because MELT would generate some C++ code, then compile and dynamically load (using dlopen) it. We'll see below how to make that faster (using the gofindgimple mode), by avoiding useless recompilation of generated C++ code.

Here are some more examples of findgimple modes uses (we omit the always needed -fplugin=melt -fplugin-arg-melt-mode=findgimple for brevity). Notice that patterns (all starting with ?) may be nested:

Notice that the findgimple mode is always generating and compiling some C++ file (for the particular search you are interested in), and that usually takes several seconds (then all the generated files are deleted). If the same search is to be done one many files (e.g. all the translation units of your software) it is worthwhile to keep the generated files. For that, use the -fplugin-arg-melt-output=file-prefix argument of the findgimple mode. Then the generated C++ files (named file-prefix*.cc for real generated code or file-prefix*.[ch] for meta-data code) and the file-prefix*.so module are kept. To reuse them again, use the gofindgimple mode like -fplugin-arg-melt-extra=file-prefix -fplugin-arg-melt-mode=gofindgimple (without giving any -fplugin-arg-melt-arg, of course).

Use the justshowpasses mode to show the passes executed by GCC. Of course, you could also (and instead) pass -fdump-all-passes to gcc.

MELT modes for evaluation

Another interesting mode is the runtime evaluator; you can evaluate some non-empty sequence of one or more MELT expressions with the eval mode, giving the sequence of expressions to evaluate thru a string passed with -fplugin-arg-melt-arg=MELT-expressions. For example,
gcc -fplugin=melt -fplugin-arg-melt-mode=eval \
  -fplugin-arg-melt-arg='(melt_version_str)' -c empty4melt.c is evaluating a sequence made of one single expression (melt_version_str) -we have to add quotes to keep the bash shell happy- and printing the result (of the last evaluated expression); you may get some warnings and an output having lines similar to:

cc1: note: MELT generated new file /tmp/fileRi68N5-GccMeltTmp-101885fb/melt_p17576_run0001_eXt.cc
cc1: note: MELT generated C++ code of module /tmp/fileRi68N5-GccMeltTmp-101885fb/melt_p17576_run0001_eXt 
           with 0 secondary files in 0 CPU millisec.
MELT is building binary /tmp/fileRi68N5-GccMeltTmp-101885fb/melt_p17576_run0001_eXt from 
           source /tmp/fileRi68N5-GccMeltTmp-101885fb/melt_p17576_run0001_eXt 
           with flavor runextend
cc1: note: MELT plugin has built module /tmp/fileRi68N5-GccMeltTmp-101885fb/melt_p17576_run0001_eXt
           flavor runextend
cc1: warning: MELT debug depth -f[plugin-arg-]melt-debug-depth= defaulted to 9 [enabled by default]
;;; result of eval mode of 1 expression[s] is ...
 "1.0-rc3 [melt-branch_revision_204086]"
;;; *** end of evaluated result *** 
cc1: note: MELT removed 7 temporary files from /tmp/fileRi68N5-GccMeltTmp-101885fb

The above (simplified) output shows that MELT has made a temporary directory, has filled it with some temporary C++ files, has compiled them into a module (some *.so file inside), has dlopen-ed that module and executed it, and finally has cleaned that temporary directory.

The MELT expression (melt_version_str) illustrates the Lisp-like syntax of the MELT language: almost every MELT expression is enclosed in parenthesis: after a left paren ( you get the operator (or functions), followed by optional operands (or arguments), ending with a right paren ) so parenthesis are highly signficant.

As an exercise, try to evaluate melt_version_str without the parenthesis. The result is the value for the melt_version_str primitive itself, it is not a string.

You probably will want to use many times the MELT expression evaluator. To avoid typing many times similar long commands, you may want to define the following bash function: 

# for interactive bash shell
function melt_eval {
  touch empty4melt.c;
  gcc -fplugin=melt -fplugin-arg-melt-mode=eval \
      -fplugin-arg-melt-arg="$*" -c empty4melt.c -o /dev/null
}

Once you have such a shell function (perhaps even in your $HOME/.bashrc if you wish) you could simply type melt_eval '(melt_version_str)'. However, your shell requires you to backslash-escape some characters, notably the ' (quote character), like melt_eval '\'1' or simply melt_eval \'1

You can also evaluate a file containing MELT expressions with the evalfile mode, giving with -fplugin-arg-melt-arg=melt-file the name of the MELT source file. If no name, or if - is given, the standard input is read till end of file, all the expressions are evaluated, and the result of the last one is printed (by translating them to some C or C++ temporary file, compiling it, and dlopen-ing the temporary shared object).

You can run a MELT file with the runfile mode. The file name should be passed with -fplugin-arg-melt-arg=melt-file, it is run (also by generating C or C++ code, compiling it, then dlopen-ing it) for its side effect without implicitly displaying any result.

There is an experimental read-eval-print loop provided by the repl mode. Expressions are grouped in sequences, terminated by a double newline, and read from stdin. So you can type MELT expressions, and when you enter an empty line (separating two sequences), they are evaluated.

Other MELT modes.

Modes to translate your MELT source to generated C or C++ are explained below.

An interesting mode (removed in MELT 1.1), was the probe (if you have installed the optional probe GUI and have a melt-probe script). It shows you, in a graphical interface window, the source code files, and enable you to get intersting GCC internal representations (notably Gimple-s and Tree-s) at a particular source location. Try something like
gcc -fplugin=melt -fplugin-arg-melt-mode=probe -O -Iyour-include -c source-file.c on a relevant source-file.c (with your-include directory) of your application. Future versions of MELT might provide a Web interface (then the probe would be deprecated).

Most of the modes above are happier with an empty4melt.c source input to gcc.

Various files and directories used by GCC MELT

The GCC MELT plugin handles various files and directories:

Conceptually, GCC MELT does not load a module, it loads a descriptive generated file (and all the files mentionned in it), or a module list, and may recompile the module[s] (of some flavor) if it is missing.

The sources path is a colon separated path of source directories. The modules path is a colon separated path of module directories.

Don't forget to give a work directory to GCC MELT. Files created there are never removed automatically, it is up to you to clean it.

MELT may create a temporary directory which will be cleaned and removed by the plugin after the compilation.

Translating your MELT source files

You often want to translate your *.melt files to MELT modules. This involves a (quick) translation of your MELT code into big generated C or C++ code, then a (slow) compilation of these generated files into a MELT module. The actual bottleneck is the compilation (with a make process started by the MELT plugin) of the generated C or C++ code into .so modules, because the emitted C or C++ code is much larger than the original MELT file (an expansion factor of more than 10x is not uncommon).

Advice: avoid giving a file name to your MELT source file similar to the files on which your MELT modules will be used. So don't have your foo.melt used to extend the compilation of your foo.c.

The MELT translator is not very friendly on MELT syntax errors. It should locate the first few errors in your *.melt file, but don't expect it to continue reliably the translation. Since the MELT → C or C++ translation runs quickly, reliable syntax error recovery don't matter much in practice.

Translating options

The following MELT modes are relevant to translation of *.melt files:

All the translating modes above require the MELT source file yourfile.melt to be given with the -fplugin-arg-melt-arg=yourfile.melt option to gcc (or g++, etc). You certainly will give an empty4melt.c input source to gcc or pass -c -x c /dev/null -o /dev/null, etc. You may give the output module with the -fplugin-arg-melt-output=module option. You usually want to give some work directory -fplugin-arg-melt-workdir=work-directory/
(because MELT may notice that it is generating an already existing C or C++ emitted file there, and won't overwrite it, so the internal make won't need to recompile it again.).

Using make to tranlate your .melt files

When using GNU make in your development process, you may want to add the following fragments to your Makefile:

Hint: if you are an Emacs user, compile inside emacs with M-x compile

Using your own MELT extensions

Your extensions usually need to define their own MELT modes. You can load an extra module with the -fplugin-arg-melt-extra=your-modules argument to gcc or g++ etc. So if you have a yourmeltextension.melt file defining some yourmode mode, you'll probably pass -fplugin-arg-melt-extra=yourmeltextension -fplugin-arg-melt-mode=yourmode to your GCC compiler. For example, you could add in your Makefile a specific rule to use yourmeltextension when compiling some-file.cc, like: 

#when compiling some-file.cc use MELT with optimized module from yourmeltextension.melt  and yourmode
some-file.o: some-file.cc yourmeltextension.optimized.so | meltworkdir
     $(MELTGXX) -fplugin-arg-melt-extra=yourmeltextension \
               -fplugin-arg-melt-mode=yourmode \
           $(CXXFLAGS) -c $< -o $@

During development of your MELT extension, you may want to run some tests. Then you probably want to enable MELT to spill numerous debug messages, in particular when you code some (debug "Here x=" x) debugging expressions. This is enabled with -fplugin-arg-melt-debugging=mode (or -fplugin-arg-melt-debug) when using quicklybuilt flavored modules. So for debugging your MELT extension you probably want to explicitly pass -fplugin-arg-melt-extra=yourmeltextension.quicklybuilt to gcc or g++ (i.e. your $(MELTGXX) in your Makefile), since the default flavor of your extension module is optimized. Notice that optimized flavor of modules won't give any debugging output (nor check your assert_msg assertions), but quicklybuilt module flavor enable such debugging output (which you should explicitly activate at run-time with -fplugin-arg-melt-debugging=mode or -fplugin-arg-melt-debugging=all).