There are many guides on H-Sphere forums about how to use open_basedir and safe_mode however they usually include cumbersome settings and alterations on H-Sphere files which are not practical. Here you can find how I solved the problem, the simple way. (Updated for H-Sphere 3.1)

Most people think that using safe_mode and open_basedir causes problems to applications creating files on the server (for example gallery or joomla) however this is not the case. Especially if you are running a FreeBSD web server. The problems witht he applications mentioned above only exist in Linux (too bad for Linux users)

Why FreeBSD is different? Because on FreeBSD the files created under a directory automatically inherit the GID of the directory. In Linux this is only the case if you set the directory mode g+s. Having the file in the same GID that the user have enables us to use the safe_mode_gid of option of the PHP which allows applications to continue working even if the file is owned by the 'httpd' user (temporarily as we will see below).

So the long waited instructions:

1- Make the necessary settings in vhost.config file to add open_basedir to the user vhost configurations. Please first read this page about vhost.config before continuing:
http://www.psoft.net/HSdocumentation/sysadmin/altering_vhost_configuration.html

You should add the following to be within VirtualHost tags. One should follow the instructions in psoft's template customization manuals for making this change permanent in this page:
http://www.psoft.net/HSdocumentation/customization/template_customization.html

Additions to be made to vhost.config, if you are using H-Sphere 3.1 you should use <IfDefine LIBPHP4> and  <IfDefine LIBPHP5> instead of IfModule statements. Note that these wont work if you are using FastCGI/CGI for running PHP on H-Sphere 3.1

<IfModule mod_php4.c>
  php_admin_value open_basedir ${hosting.path}:/tmp:/hsphere/shared/apache/libexec/php4ext
</IfModule>
<IfModule mod_php5.c>
  php_admin_value open_basedir ${hosting.path}:/tmp:/hsphere/shared/apache/libexec/php5ext
</IfModule>

Here is a diff of 2.5p9 vhost.config file and the modified one. (text might be wrapped)

38a39,46
>
> <IfModule mod_php4.c>
> php_admin_value open_basedir ${hosting.path}:/tmp:/hsphere/shared/apache/libexec/php4ext
> </IfModule>
> <IfModule mod_php5.c>
> php_admin_value open_basedir ${hosting.path}:/tmp:/hsphere/shared/apache/libexec/php5ext
> </IfModule>
>
79a88,96
>
> <IfModule mod_php4.c>
> php_admin_value open_basedir ${hosting.path}:/tmp:/hsphere/shared/apache/libexec/php4ext
> </IfModule>
> <IfModule mod_php5.c>
> php_admin_value open_basedir ${hosting.path}:/tmp:/hsphere/shared/apache/libexec/php5ext
> </IfModule>
>

2- The next step is enabling safe_mode. This is easily accomplished by enabling it in php.ini. The following variables should be set.

safe_mode = On
safe_mode_gid = On
safe_mode_include_dir = /hsphere/shared/apache/libexec/php5ext/php
safe_mode_exec_dir = /usr/local/safebin

You also will need to set upload_tmp_dir directive. Although the default is /tmp this helps uploads to be able to read from /tmp directory.

upload_tmp_dir = /tmp 

You should use php4ext directory if you are using php4 of course. Also create a /usr/local/safebin directory which can be accessed by 'httpd' user (you can decide what ownership and permissions can this have as there are multiple ways to make this work). Then put binaries required by php scripts in use to this directory as php programs won't be allowed to run system binaries laying around anymore.

3- Setup a script to fix permissions of uploaded files (by php). You can find the script I am using from my web site at http://dev.yurtesen.com/component/option,com_remository/Itemid,26/func,fileinfo/id,18/

It changes files to user:httpd mode 770 and directories to user:user mode 777. One might ask, why not change the directories to user:httpd and 770 as httpd can still write to this directory. But doing this creates a chicken egg problem. If php creates a file inside a directory owned by user:httpd the file would ahve httpd:httpd ownership. Where a script with user:user can not access to this file anymore! Of course if you are using Linux this is the case whatever you do unless you set the directory mode to g+s which is not good news for Linux users. On FreeBSD setting the directory to user:user causes files to be created as httpd:user which is what we want. Of course the script is setting g+s to directories created by php in Linux so it tries its best but Linux is really troublesome on this area.

If you are thinking that there are other scripts similar to this around. You are not quite right. This script behaves differently on directories and files and it even behaves differently on directories depending on if you have Linux or FreeBSD and tries to set the best permissions as explained above.

Please let me know if there are any mistakes in this documentation.